Important !!!
New IP ADRESSES After Migration to Azure
With the migration of our FTP servers to Azure 2 new IP Adresses have to be whitelisted on the client’s environment

These IP adresses are

• TNI: 4.185.2.199

• PRD: 131.189.106.246

In addition to offering web services, MAGDA also provides file exchange. The exchange of data occurs via FTP. These are secured with FTPS, where all information is encrypted and sent over a TLS channel, using a VO-DCBaaS certificate.

The exchange of files between users of the MAGDA platform and MAGDA occurs via FTPS servers. This means that the user's application prepares or retrieves files from one of the VIP FTP Servers using a so-called FTP client that supports FTPS. MAGDA itself does not place files on the servers of users and does not pick up files from the users.

It is the responsibility of the user (application) to timely retrieve the files from the VIP FTPS server, taking into account the guidelines regarding the maintenance of those folders.

Open

Before a user can use file transfer services from MAGDA and connect to the MAGDA platform, the user must have gone through an admission request on the MAGDA platform. During the admission procedure, the user will be assigned their own home folder. The home folder of each user is by default divided into the following subfolders.

to_vip	Here, the user/application will prepare the request files for VIP. After processing, VIP removes the files from this folder (and places them in cache\to_vip).	Open

to_vip	Here, the user/application will prepare the request files for VIP. After processing, VIP removes the files from this folder (and places them in cache\to_vip).	Open
from_vip	Here, VIP places all files for the user/application, including both response files and (non-public) publications. The user will take the initiative to retrieve these files and is responsible for maintaining this folder; in other words, the user/application will delete the already retrieved files themselves. Optionally, the retrieved files can be transferred to cache\from_vip before deletion.
cache	The cache folder has 2 subfolders, to_vip and from_vip.
cache/to_vip	Contains all files that have been picked up by VIP. This allows selective files to be put back into the processing chain within the retention period.
cache/from_vip	Contains all files that have been picked up by the application and then moved to this folder.
public	Contains all publications intended for everyone, i.e., not specifically for users/applications.

When the user/application places such a request file in the FTP server's to_vip folder, MAGDA will detect that file within a certain time (based on a predefined naming pattern), pick it up, and process it. The file will first be copied to the cache/to_vip folder and then passed on to the MAGDA processing environment.

Information about using web services in batch and preparing request files in the context of batch queries can be found at Werking webservice in batchPreview.

Response files created in the context of batch processing are placed by MAGDA in the 'From_vip' folder where they can be retrieved by the user.

Information about using web services in batch and the formatting of response files in the context of batch queries can be found at Werking webservice in batchPreview.

Through MAGDA, data files from authentic sources are made available to users. Publications exist in both XML format and CSV format, but this depends on the type of publication. Reference is made to the service-specific documentation. Often, only XML is supported.

Public: There are a number of generic publications (mainly business mutations and code tables), which are made available by default to all regular FTP users. These generic publications are placed in the public folder.

from_vip: All other publications, including custom extracts, initial loads, and privacy-sensitive publications of personal data, are placed by MAGDA in the from_vip folder of the entitled user. The frequency and timing depend on the type of data.

Users are expected to timely pick up the publication files. Regarding user-specific publications placed in the from_vip folder, users are expected to delete them as soon as they are picked up, possibly after first placing a copy in cache/from_vip.

The generic publications (in the Public folder), on the other hand, cannot be deleted by the user but are automatically removed by MAGDA after a retention period of 1 month. Therefore, there is no reason/need to cache the retrieved files in cache\from_vip.

For the naming of both the files to be delivered and the files offered by MAGDA, reference is made to the service manuals: Handleidingen MAGDA-diensten per domeinPreview

As previously mentioned, MAGDA relies on users to maintain their folders (from_vip and cache\from_vip). Users are encouraged to provide automated scripts that copy the file to the cache immediately after retrieving a file and then delete it from the from_vip folder.

However, MAGDA takes into account that not all users will always follow these guidelines. For this reason, MAGDA will regularly clean up the folders by removing old files. Below is a summary of the various folders that exist within a user account, along with a brief description of their maintenance.

As previously described, a user must always go through an admission procedure and obtain an FTPS account to exchange data with MAGDA. Because access control based on account (with username and password) offers little protection against unauthorized use, MAGDA requires that the connection be encrypted and that users and their applications be identified using a certificate.

To protect the MAGDA servers from external attacks, all FTPS communication must occur from a pre-designated PC or server on the user's side. Therefore, during the connection process, the user must provide the IP address of the PC or server that will be used for FTP communication. At the firewall level, FTP traffic will only be allowed for that IP address.

Both FTP and FTPS are based on the client-server model that characterizes other parts of the internet as well. Client software (an FTP client), installed on the user's PC/server, establishes a connection with the FTP server on the other side of the 'line'.

The difference lies in the fact that for FTP, one works with a username and password to log in. With FTPS, a client certificate is used to log in. Here, the CN value of the certificate is used as the username, but no password is required. The certificate serves as the password.

Client certificates used for FTPS follow the same standard as certificates for Web services.

• The CN that must be included in the certificate will be determined by MAGDA and made available in user management and the VO-DCB application (for the target group of VO-DCB, users).

• The user creates a Certificate Signing Request (CSR) for the CN.

• The user generates the certificate with VO-DCB based on the CSR.

• The user logs in using the certificate.

On Windows platforms, the FTP client WinSCP is the best client for working with FTPS servers that require a client certificate. WinSCP is a free program.

On Linux platforms, the command line tool lftp is available.

When connecting to the FTPS server for the first time, you may receive a message with a warning about an "unknown" server, asking for your consent to proceed. This stores the server certificate on the user's platform (PC or server).

Once the host key is accepted/stored, the rest of the communication can continue. In subsequent connections to the same FTPS server, the host key will already be known, and the above confirmation screen will no longer appear.

The FTPS servers have the following DNS names where you can reach them:

For FTPS connections, two ports are used: a command port and a data connection port.

The FTP protocol uses port 21 for commands by default. The FTPS version on the MAGDA platform uses the same port (FTPS Explicit).

The data connection ports are fixed in the range of 40000-42000.

This means that users behind a forward proxy must whitelist ports 21 OUTBOUND and 40000-42000 INBOUND to access the new FTPS server.

In WinSCP, the following configuration must be made:

Open

‌

Creating a new site:

• File Protocol:      FTP

• Encryption:         TLS/SSL Explicit encryption

• Hostname:          vipftps.vlaanderen.be for production, or vipftps-aip.vlaanderen.be for testing

• Port number       21 (Default do not change)

• Username           Must match the CN of the certificate being used

Then click on the "Advanced" button to set the certificate and private key:

Open

In "Advanced" - "TLS/SSL" tab:

• Minimum TLS/SSL version: set to TLS 1.2

• Client certificate file:  point to the certificates and private keys

For the certificate and private keys, we strongly recommend using the PKCS12 format. The private key is also then secured by a password that must be provided during the creation of a PKCS12 keystore. See also Certificates: Creation on this user environment.

PKCS12 keystores can be created by using the Java "keytool IUI" program or by using the OpenSSL program:

openssl pkcs12 -export -in certificate.pem -inkey key.pem -out cert_and_key.p12

If the previous command is used, a password will be requested for the key (if needed), and another password will be requested for the resulting file.

After pressing the "Login" button, a password will be requested for the ".p12" file. Here, the password used for creating the .p12 file must be entered.

For other operating systems (Linux, UNIX), you can use Curl or LFTP to make use of FTPS 2-way. Below you will find the necessary commands to establish the connections. Of course, the same ports must be open as described above (port 21 as the control port and all ports between 40,000 and 42,000 through which the data will be sent).

• Curl
  With Curl, it is possible to make a FTPS 2-way connection. You must install Curl on your server.
  Below you will find the syntax for connecting to our TNI environment:
  curl -v --sessionid -u usernameofthecertificate:blabla --ftp-ssl --cert locationofyourcertificatefile/crt file of the certificate
  --key locationofthekeyfileofyourcertificate/_keyfileofthecertificate --pass passwordofyourcertificateinplaintext _--cacert vipftps-aip_vlaanderen_be.crt -p -x ftp://vipftps-aip.vlaanderen.be/from_vip/* -l
  Below you will find the syntax for connecting to our PRD environment:
  curl -v -u usernameofthecertificate:blabla --ftp-ssl --cert locationofyourcertificatefile/crt file of the certificate
  --key locationofthekeyfileofyourcertificate/_keyfileofthecertificate _--cacert vipftps_vlaanderen_be.crt ftp://vipftps.vlaanderen.be/from_vip/* -l

  Legend:
  usernameofthecertificate:blabla : your username is the same as the CN of your certificate. The colon and blabla is because curl always expects a password even though it is not used in this case (your certificate is used as the password). You may use any password as long as your certificate is correct.
  locationofyourcertificatefile/crt file: here you must specify the location where you have saved your certificate as well as the crt file of your certificate.
  locationofthekeyfileofyourcertificate/keyfileofthecertificate: you must also specify the location of the key file of your certificate as well as the key file itself.
  vipftps-aip_vlaanderen_be.crt/vipftps_vlaanderen_be.crt : these are the server certificates that you specifically need to trust. If necessary, we can provide those to you.

• LFTP
  LFTP is a tool that allows you to make FTPS 2-way connections. Of course, this must be installed on your server.
  You will notice that LFTP will ask for a password (even if you specify a certificate). You can just press Enter, and LFTP will use the certificate as the password. If it is the correct certificate, the connection will succeed.

  Below you will find the syntax for connecting to TNI:

‌

‌

‌

!!!!CURL and LFTP VIA PROXY!!!!

Some users will need to connect to our vipftps servers via a proxy. This is a complex situation because not all proxy technology supports FTPS 2-way. If you are sure that your proxy supports FTPS 2-way, you will find below the commands to connect with Curl or LFTP over a proxy.

CURL

For TNI

curl -v --sessionid -u usernameofthecertificate:blabla --ftp-ssl --cert locationofyourcertificatefile/crt file --key locationofthekeyfileofyourcertificate/_keyfileofthecertificate _ --pass passwordofyourcertificateinplaintext --cacert vipftps-aip_vlaanderen_be.crt -p -x http://proxyserver:proxypoort ftp://vipftps-aip.vlaanderen.be/from_vip/* -l

For PRD
curl -v --sessionid -u usernameofthecertificate:blabla --ftp-ssl --cert locationofyourcertificatefile/crt file --key locationofthekeyfileofyourcertificate/_keyfileofthecertificate _ --pass passwordofyourcertificateinplaintext --cacert vipftps_vlaanderen_be.crt -p -x http://proxyserver:proxypoort ftp://vipftps.vlaanderen.be/from_vip/* -l

Legend

locationofthekeyfileofyourcertificate: this is the location of and the key file itself of your certificate that you need to specify.
usernameofthecertificate: your username is the same as the CN of your certificate.
passwordofyourcertificateinplaintext the password you have on your certificate. You need to type this in plain text.

proxyserver:proxypoort: this is the name of your proxy server followed by a colon and the port through which the requests will go (the default is 8080, but it can be another. Ask your network people which one is used).

LFTP
For TNI

set ftp:proxy http://proxyserver:proxypoort
set hftp:Proxy http://proxyserver:proxypoort
set hftp:use-type false
set ftp:use-hftp false
set ftp:ssl-auth TLS
set ftp:bind-data-socket true
set ftp:ssl-force true
set ftp:ssl-protect-data true
set ftp:ssl-protect-list true
set ftp:ssl-data-use-keys true
set ftp:ssl-copy-sid true
set ftps:initial-prot "P"
set ftp:use-allo false
set ssl:verify-certificate true
set ssl:cert-file yourcrtfile
set ssl:key-file thekeyfileofyourcertificate
set ssl:check-hostname true
set ftp:passive-mode true
set ftp:use-site-utime2 false

open -u usernameofthecertificate,"" ftp://vipftps-aip.vlaanderen.be
Legend
yourcrtfile : this is the certificate you use to log in.
thekeyfileofyourcertificate: this is the key file of your certificate that you need to specify.
usernameofthecertificate: your username is the same as the CN of your certificate.
For PRD

set ftp:proxy http://proxyserver:proxypoort
set hftp:Proxy http://proxyserver:proxypoort
set hftp:use-type false
set ftp:use-hftp false
set ftp:ssl-auth TLS
set ftp:bind-data-socket true
set ftp:ssl-force true
set ftp:ssl-protect-data true
set ftp:ssl-protect-list true
set ftp:ssl-data-use-keys true
set ftp:ssl-copy-sid true
set ftps:initial-prot "P"
set ftp:use-allo false
set ssl:verify-certificate true
set ssl:cert-file yourcrtfile
set ssl:key-file thekeyfileofyourcertificate
set ssl:check-hostname true
set ftp:passive-mode true
set ftp:use-site-utime2 false

open -u usernameofthecertificate,"" ftp://vipftps.vlaanderen.be
Legend
yourcrtfile : this is the certificate you use to log in.
thekeyfileofyourcertificate: this is the key file of your certificate that you need to specify.
usernameofthecertificate: your username is the same as the CN of your certificate.

Users who want to establish a connection to our FTPS servers with an application must follow roughly the same steps as for a UI client:

• Use an FTP library that supports FTPS client certificates for the programming language being used;

• Create a keystore containing the client certificate and the private key;

• Optionally, create a trust store, which will contain the VO DCB Root CA certificate;

• Set up the necessary network connections.

Here is a page that describes in more detail how the libraries should be used: FTPS Library Technical Information

‌

Since we work with FTPS 2-way (where both your certificate and our FTP server's certificate are checked), our server certificate is updated periodically. You will then need to trust our server certificate in your trust store from time to time. You will be notified in advance so that you can take the necessary steps for this.
To renew this server certificate, the steps to be taken differ depending on how you trust our server certificate. You can trust our server certificate at the highest level by adding the Root Certification Authority (RCA) and Intermediate Certification Authority (ICA) to your trust store. The Certification Authorities can be found at https://documenten.pki.vlaanderen.be. This concerns the Certificate Flemish government Root CA 2020 and Certificate Flemish government Issuing CA 2 2020 (see screenshot below)

Open

Open

Additionally, you can trust our server certificate at the lowest level by requesting the server certificate via OpenSSL. You must have OpenSSL installed for this. The commands to obtain the server certificate are below.
For TNI

openssl s_client -connect vipftps-aip.vlaanderen.be:21 -starttls ftp
For PRD

openssl s_client -connect vipftps.vlaanderen.be:21 -starttls ftp

‌

The output of both commands will display the server certificate. It starts with -----BEGIN CERTIFICATE----- and ends with -----END CERTIFICATE-----
This entire piece of output (including the first dash to the last dash) must be saved as a .crt file and added to a trust store or used in your command to connect to vipftps-aip.vlaanderen.be or vipftps.vlaanderen.be (e.g., in the cacert part of Curl as mentioned above).

‌

We have noticed that users who have been connecting with FTPS 2-way for a long time and have downloaded their certificate a long time ago and upgrade to a new OpenSSL may receive an error message that reads as follows:
digital envelope routines::unsupported

The cause is that the new OpenSSL engine does not (or no longer) support the encryption algorithm that encrypts the certificate.
The solution is to download the certificate again from https://dcb.vlaanderen.be

More information about this error message when using Winscp can be found here
https://winscp.net/forum/viewtopic.php?t=34236

More information about this error message when using scripting for the connection via Curl, JS, and others can be found here
https://stackoverflow.com/questions/72598983/curl-openssl-error-error0308010cdigital-envelope-routinesunsupported