Flat file configuration issue (storageQuotaExceeded) in "Metadata Flat File Syncer tool"

Support - Assistance technique
1

Hello,

I’m trying to configure the dhis2-metadata_flat_file_syncer tool but after following all the steps in the README file, I can’t get it to work. Once the backend is running and I open templates/index.html in my browser to export the metadata to a flat file, I keep getting the same error:
“An error occurred: {‘code’: 403, ‘message’: “The user’s Drive storage quota has been exceeded.”, ‘errors’: [{‘message’: “The user’s Drive storage quota has been exceeded.”, ‘domain’: ‘usageLimits’, ‘reason’: ‘storageQuotaExceeded’}]}”

I have read a lot of documentation about service accounts to make it work with my personal storage on Drive but I know that other users didn’t need to configure anything else.

Does anyone have any idea why this might be happening?

Thank you in advance.

2

Hi @pma

Welcome back to the DHIS2 Community of Practice! :clap:

I believe that the tool you are referring to is (dhis2-utils/tools/dhis2-metadata_flat_file_syncer at master · dhis2/dhis2-utils · GitHub); however, please know that the more official ways for metadata import/export are either the Import/Export app the Web API, or well known community apps in the App Hub such as the Metadata Syncronization.

Please double check your Google Drive storage space because this issue shows that your account no longer has enough storage space which the app needs in-order to update the Google Sheets.

Thanks!

3

Hi @Gassim, thank you very much for your quick response. I appreciate your suggestions, but this flat file syncer tool is more suitable for what I need at the moment, which is to upload a large amount of metadata from an Excel file.

The point is that I don’t control or have access to the Drive storage for my service account, then I’m supposed to redirect (“impersonate”) it to my personal Drive storage (I have read that storage in service accounts is 0, which is why I’m getting that error). But I haven’t found anything about this in the documentation.

1 Like
4

If anyone else encounters the same problem, the only solution I have found is to use OAuth2 with a personal account, i.e., replace ServiceAccountCredentials with OAuth2 with user consent (creating credentials of type “OAuth 2.0 Client ID” and application type “Desktop”).

After that, replace the authorize_google_sheets() function with:

from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
import pickle

def authorize_google_sheets(auth_file):
SCOPES = [‘``https://www.googleapis.com/auth/spreadsheets’``, ‘``https://www.googleapis.com/auth/drive’``]

creds = None
token_file = 'token.pickle'

if os.path.exists(token_file):
    with open(token_file, 'rb') as token:
        creds = pickle.load(token)

if not creds or not creds.valid:
    if creds and creds.expired and creds.refresh_token:
        creds.refresh(Request())
    else:
        flow = InstalledAppFlow.from_client_secrets_file(auth_file, SCOPES)
        creds = flow.run_local_server(port=0)

    with open(token_file, 'wb') as token:
        pickle.dump(creds, token)

try:
    return gspread.authorize(creds)
except Exception as e:
    print(f"Error when authorizing with Google Sheets: {e}")
    exit()
1 Like
5

Hi @pma
Great to see that you have found the solution, and thank you for sharing with the community.

It looks like it was trying to use the service instead of a user account so there was no issue in the drive storage :+1: