vigneshss-07 / cloud-ai-analytics

This Repo contain details related to Data Engineering tech stacks in GCP

Hey there..!!! I'm Vignesh S S!

• Hi, I'm a passionate Data Engineer from India. My passion for data lies with dreaming up ideas and making them come true with elegant ETL/ELT and flexible bigdata pipelines. I take great care in models, pipelines and code quality of the things I build.
• I am also an open-source enthusiast. I learned a lot from the open-source community and I love to collaborate and share knowledge.
• I’m currently working as a Data Engineer in H&M.
• Completed Google Certified Professional Cloud Architect Professional Cloud Architect
• Completed Google Certified Professional Cloud Database Engineer Professional Cloud Database Engineer
• Completed Google Certified Professional Machine Learning Engineer Professional Machine Learning Engineer
• Completed Google Certified Professional Cloud Developer Professional Cloud Developer
• Completed Google Certified Professional Data Engineer Professional Data Engineer
• Completed Google Certified Associate Cloud Engineer Associate Cloud Engineer
• Completed Google Certified Cloud Digital Leader Digital Leader
• Completed Microsoft Certified Azure Data Fundamentals Microsoft Azure Data Fundamentals
• Completed Microsoft Certified Azure Data Scientist Associate Microsoft Data Scientist
• Completed Microsoft Certified Azure AI Engineer Associate Microsoft Azure AI Engineer
• Completed Certification in Microsoft as Microsoft Certified Azure AI Fundamentals Microsoft Azure AI Fundamentals
• Completed Certification in Microsoft as Microsoft Certified Azure Fundamentals Microsoft Azure Fundamentals
• Completed Certification in Databricks as Academy Accreditation - Databricks Lakehouse Fundamentals Databricks Lakehouse Fundamentals
• Completed Certification in Databricks as Academy Accreditation - Platform Administrator Platform Administrator
• Completed Certification in Microsoft as Microsoft Certified Security, Compliance, and Identity Fundamentals Security, Compliance, and Identity Fundamentals
• Completed Certification in Microsoft as Microsoft Certified: Dynamics 365 Fundamentals (CRM) Dynamics 365 Fundamentals (CRM)
• Completed Certification in Amazon as AWS Certified Cloud Practitioner AWS Cloud Practitioner
• Completed Certification in Microsoft as Microsoft Certified: Power Platform Fundamentals Power platform Fundamentals
• Completed Microsoft Certified: Azure Network Engineer Associate Microsoft Azure Network Engineer Associate
• Completed Microsoft Certified: Azure Enterprise Data Analyst Associate Microsoft Azure Enterprise Data Analyst Associate
• Completed Microsoft Certified Azure Security Operations Analyst Associate Microsoft Azure Security Operations Analyst Associate
• Completed Microsoft Certified Identity and Access Administrator Associate Microsoft Identity and Access Administrator Associate
• My previous employer is TCS-digital where my primary tasks is to build various ML POC under airlines and retail domains.
• Actively writing blogs on Linkedin in Cloud & AI Analytics
• Sharing back to community through youtube on advanced techniques in cloud like Azure, Google Cloud Platform, SQL, Statistics, ML and NLP related technologies through Cloud & AI Analytics.
• Actively learning and looking for better opportunities in Machine Learning and Bigdata technologies.
• Open for collaborations in Cloud Platform, Data Science, Data Engineering and Machine Learning projects.

-> Scroll down to see my Pinned Repositories.

def vignesh_07():
  skills_with_proficiency = {'Python': 'Advance', 'PySpark' : 'Advance','SQL & NoSQL': 'Intermediate', 'DSA': 'Intermediate', 'Statistics' : 'Advance'}
  tools = ['dbt tool', 'github', 'BigQuery', 'sklearn', 'Data Modeling', 'Data Analytics', 'json', 'yaml', 'powerBI', 'pyspark', 'Looker Studio', 'GCP Analytics Services', 'Bigdata technologies', 'etl/elt pipelines', 'ci/cd-devops']
  cloud_architecture = [' Google Cloud Platform', 'Microsoft Azure Services']
  active_blogger = True
  return cool

📕  Latest Blog Posts

• Integrate Azure Databricks with Azure Machine Learning Workspace
• Create Google Cloud Storage Bucket Using UI In Google Cloud Platform
• Create Google Cloud Storage Bucket Using CLI In Google Cloud Platform

Connect with me:

Note: Please do not spam on the above mentioned profiles.

cloud-ai-analytics's Issues

Outlet_Type is at 11th Position in list

HI Vignesh,
Outlet_Type is at 11th Position in the list so pass 10 in place of 8 will produce the correct result. Thanks

bigquery_client in BigQuery/bigquery.py does not appear to be defined

Hi there. Followed one of your YouTube tutorials on using API in Python to create a dataset and table which you provided here on your repo

In the provided Python code, the variable bigquery_client does not appear to be defined in the code itself. The functions bq_create_dataset and bq_create_table are defined to take a client parameter, but the actual variable name bigquery_client is not used within the functions. I think it could also be missing 'client = bigquery.Client()' but again - I may be misunderstanding.

Greatly appreciate any insight you can provide.

#import libraries
from google.cloud import bigquery
from google.cloud.exceptions import NotFound


#Function to create a dataset in Bigquery
def bq_create_dataset(client, dataset):
    dataset_ref = **bigquery_client**.dataset(dataset)

    try:
        dataset = **bigquery_client**.get_dataset(dataset_ref)
        print('Dataset {} already exists.'.format(dataset))
    except NotFound:
        dataset = bigquery.Dataset(dataset_ref)
        dataset.location = 'US'
        dataset = bigquery_client.create_dataset(dataset)
        print('Dataset {} created.'.format(dataset.dataset_id))
    return dataset

#Function to create a dataset in Table
def bq_create_table(client, dataset, table_name):
    dataset_ref = bigquery_client.dataset(dataset)

    # Prepares a reference to the table
    table_ref = dataset_ref.table(table_name)

    try:
        table =  bigquery_client.get_table(table_ref)
        print('table {} already exists.'.format(table))
    except NotFound:
        schema = [
            bigquery.SchemaField('S_No', 'INTEGER', mode='REQUIRED'),
            bigquery.SchemaField('Age_in_cm', 'INTEGER', mode='REQUIRED'),
            bigquery.SchemaField('Weight_in_Kg', 'INTEGER', mode='REQUIRED'),
            bigquery.SchemaField('Name', 'STRING', mode='REQUIRED'),
        ]
        table = bigquery.Table(table_ref, schema=schema)
        table = bigquery_client.create_table(table)
        print('table {} created.'.format(table.table_id))
    return table