This project contains scripts to extract metadata from DICOM files, NIFTI files and spreadsheets, and import them into a database.

ETL process applied on covid-19 dataset of European countries using Azure services such as databricks, keyvault, sql database, data factory etc. Finally power bi dashbaord was also made.

Azure Data Factory Integration Runtime in Windows Container.

Generate a flat CSV file from an I2B2 data (containing brain features, demographic data, etc).

An extensible test data builder for any databases ，任意数据库测试数据生成

Data engineering project using Data Lake Gen2, Data Factory, Databricks (PySpark, Spark SQL) and Power BI.

Airflow Azure ETL process

Migration scripts for the Data Catalog database used to capture provenance information on all files

Merge individual MIP formatted output from the NMM pipeline into one single CSV file.

Implemented enhanced analytics for trip transactions and ride-based data from SQL Server using Medallion Architecture with Azure Data Factory and Databricks.

This Azure Function App will take a HTTP response and write it to Blob storage. This is useful in Data Factory when the API Service requires a session cookie for authentication. Today, the cookie that is obtained in a login Web Activity cannot be added to the request in a Copy Data activity. Use this approach to write data from a web call to an …

Built a data pipeline with Azure Databricks and Azure Data Factory using Formula 1 data

Migration scripts for the Data Catalog database used to capture provenance information on all files

Reorganize DICOM files following a hierarchy based on meta-data fields

Generate a flat CSV file from an I2B2 data (containing brain features, demographic data, etc).

Palak's projects

end to end data analytics engineering project usind azure and databricks

Data validation checks and sorting mechanism to efficiently handle incoming data on Azure.

Import EHR data to I2B2.