The Easy way to convert XML to JSON / JS Object and then to CSV file in Node.js
Technology is fast changing how businesses operate, and data analysis, skills, and knowledge is going to have a significant impact on how you use technology to run your business.
Every company have realized the importance of insightful data , which gives a competitive edge. There are many formats to store data . But Triad which dominate today are CSV ,JSON ,XML .In this article I am going to share the easy way to work with all of them using Node.js
While working on a project I came across a requirement to convert XML analytics dump to CSV on Serverless i.e. Using AWS Lambda and API gateway. Sounds easy right but there is heck lot of things in between .
The first and foremost challenge which I faced was File size limit allowed by AWS Lambda+API Gateway which is only 10 MB.Then How to upload files which are larger than that?
There comes the S3 bucket to the rescue. Amazon Simple Storage Service (Amazon S3) is an object storage service which allows user to store files of almost any size.Let’s go through the flowchart to get better understanding the architecture front for the API’s.
Let me give a walkthrough of Flow chart’s-
To store xml and csv file we need 2 AWS S3 buckets.So let’s create them
Create a Lambda to generate pre signed URL from S3 bucket
Create another Lambda , which will act as Event listener i.e. Will react to file upload to First S3 bucket (In my case i have named it as xmlBucket — -You can name it whatever you want provided it is unique.)
What is important in all of this is AWS Role — — Attach same role to both Lambda’s i.e default role which gets created while creating Lambda + custom s3 Trigger Role (Attach following policies to it — — sharing it as JSON)
Hit the getPresignedUrl Lambda , You will get uploadUrl,downloadUrl,fileName as response.Make a PUT request to the uploadURL with a file as binary selected in body section of Postman.
Paste the uploadUrl link as is next to PUT , it will assume header’s automatically.
Voilla, The converted csv file appears in csvStore-Bucket.Hit the download url from browser which you got in first get request to getPresignedUrl…