CDDIS https Archive Access/File Download FAQ
- How do I download data now that anonyous, unencrypted ftp was discontinued on October 31, 2020?
The CDDIS has provided examples of downloading data using other methods.
- What is the new CDDIS file download procedure?
The CDDIS changed to an https-based method to retrieve files from its archive; users can also use ftp-ssl to access the CDDIS archive. In addition, the new system uses a NASA single sign-on system for user authentication.
- Why did CDDIS discontinue traditional unencrypted, anonymous archive access on October 31, 2020?
US Government security requirements no longer allow CDDIS to offer data via traditional unencrypted, anonymous ftp. ftp, with its clear text username and password, is just not acceptable in this day and age. ftp protocol also has the disadvantage of being a two-port protocol along with both an active and passive operation mode. This means that EVERY irewall/switch in the network path between CDDIS and the data provider must be configured correctly to fnction reliably. Unfortunately, we have found too often this is not the case and, in most instances, is outside the control of CDDIS or the data provider to fix.
- What are the advantages of https over anonymous ftp?
The https protocol is as efficient as ftp transfer without the firewall/router issues of ftp. Since ftp is a two-port protocol, users can have connectivity problems (e.g., with firewall, DNS, etc.); https is a one-port protocol with fewer issues with downloads. The Earthdata Login system will be used for access through https. Earthdata Login allows users to easily search and access the full breadth of all 12 EOSDIS DAAC archives. Earthdata Login will also allow CDDIS to know our users better which will then allow us to improve CDDIS capabilities.
- Why not use sftp or some other ftp variant?
CDDIS looked into using some other variants to ftp for file upload but there were disadvantages (some major) to all of these approaches. The http(s) protocol offered more advantages and had performance equal to ftp.
- What is Earthdata Login?
Earthdata login (https://urs.earthdata.nasa.gov/) is the single mechanism for user registration and profile management for all Earth Observing System Data and Information System (EOSDIS) components. CDDIS, as one of the 12 EOSDIS Distributed Active Archive Centers (DAACs), is moving to be more integrated with its larger parent organization.
- When will this new file download system become operational?
The new file download procedure is now fully operational.
- Will my old scripts stop working?
Yes, scripts based on anonymous ftp will not work after October 31, 2020. The CDDIS has provided examples of downloading data using other methods.
- Do I need to only use cURL?
No, you can use any tool that allows you to submit an https request and that follows an http redirect (302 request). You can also use ftp-ssl. The CDDIS has provided examples of downloading data using other methods
- What browsers are supported for the web-based interface?
The latest versions of Firefox, Chrome, Internet Explorer (IE), and Safari are officially supported. The process may work in other browsers, but is not guaranteed.
- What is the index.html file found in the archive directories?
An index.html file is required in order for web browsers to be able to display the contents of a directory in a proper format. This file is used when accessing the CDDIS archive through https, not ftp-ssl. There is no harm in downloading this file; however, if you prefer to not retrieve the index.html file, you will need to modify your scripts to exclude any file named 'index.html' when downloading all files from a particular directory.
- Who do I contact for questions?
Any questions related to the new upload system should be directed to support-cddis@earthdata.nasa.gov.
- How do I download files from the https archive using the command line or a script?
Instructions and several examples of downloading files from the https archive using cURL, Wget, ftp-ssl, and Python are provided on our CDDIS Archive Access page.
- Why do I only get HTML code when trying to download a file via HTTPS?
If you do not pass your credentials properly you are automatically redirected to the Earthdata Login web page. Since you are making an HTTPS request, the server is assuming you are using a web browser and returns the login page. This normally occurs when your .netrc file is missing or in the wrong place, or you have the wrong credentials in your .netrc file.
- Why do I get strange errors when I try to change my HTTPS or plain ftp request into one using ftp-ssl?
The URL for using ftp-ssl is different from the ones used by our old ftp server and that of the HTTPS server. Please change your requests to reflect the new URL -- ftps://gdc.cddis.eosdis.nasa.gov/.
- Why am I getting an error stating that the maximum number of redirects has been followed?
You are missing the required cookie file for an HTTPS download. If your login information is not saved in a cookie file, the client is stuck in a loop of trying to login and never saves that information. Check your syntax and make sure you are setting a cookie file.
- Why am I getting a cURL (35) error when I attempt to download?
The most common reason for this is that you are using an older version of cURL that does not support the Earthdata Login requirement of TLS 1.2. You will need to update your cURL client.
Also, some very recent versions of cURL and the most recent Debian-based linux distributions have this same issue. We are working on a change on the server to fix this problem. In the meantime, please try this workaround for this issue - add this to the command line:
--ciphers DEFAULT@SECLEVEL=1 - Why aren't my credentials being read from my .netrc (or _netrc on Windows) file in my home directory?
Some users have experienced this issue; the best workaround we have found is to use the '–netrc-file' option and follow that with the full path to your _netrc file.
- I just updated to the latest Debian-based linux distribution and now I get the following error: unsafe legacy renegotiation disabled. Why?
This is a known issue at this time, the only workaround of which CDDIS is aware is to add the following lines to the end of the file /etc/ssl/openssl.cnf.
openssl_conf = openssl_init
[openssl_init]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
Options = UnsafeLegacyRenegotiation
Related Links: