Hello,
Is there a way to push and pull from Google Drive? I’m not sure whether there is an adaptor for this, but could this be achieved using http or something else?
Oh, I love this @djohnson !
There isn’t an adaptor yet, but you could use HTTP or build one based on the GoogleSheets Adaptor, as it make suse of the well supported googleapis package.
Either way (whether using HTTP or your newly created GoogleDrive adaptor)you’d want to create a new Oauth2 client that includes the right scopes to handle authentication.
I’d be happy to provide some guidance on the development of the adaptor, or if you’d like to have it done for you we can check on the availability of the OpenFn core team–or maybe one of our certified OpenFn developer partners—and see if they’re up for taking it on as paid work!
Let me know what you think and thanks for bringing this up. We’ve got plenty of people interacting with storage via SFTP and via Microsoft Sharepoint, so it would make sense to support GoogleDrive as well!
Ah, and @mtuchi , @jclark , would there be any potential “gotchas” we’d want to look out for with respect to handling large files/binaries here if we were to build a new adaptor for this?
Memory overheads for larger files feels like an obvious area of concern here. But if you’re dealing with files in the order of 10/20mb it should be fine. I don’t know what the limit would be off the top of my head.
The trick with the adaptor is to use streaming APIs are much as possible and avoid serialising large files to state. It would probably be wise to import re-export the http adaptor functions so that you can pipe file streams from drive to/from other http requests directly within a job - that’s how to keep memory usage under control.
sheets and soon gmail will be good adaptors to look at (particularly for oauth). Worth looking at msgraph too, which also has to deal with large files.
Thanks for the responses and links. I will give http a try. My JavaScript probably isn’t good enough to create an adaptor just yet. If I get it to work, I can post what I come up with.
1 Like
@djohnson I logged this as a Feature Request to get in into our backlog. Please leave a comment there with details on your use case/ a summary of what functions you want supported for us to consider in the v1 of this implementation.