Salvatore Pellitteri @pellittsa Upload data on Azure SQL Database using Azure Messaging, Stream Analytics & Logic Apps Salvatore Pellitteri @pellittsa https://pellitterisbiztalkblog.wordpress.com/

Sponsors

Organizers getlatestversion.it

Salvatore Pellitteri | @pellittsa Developer Team Manager in Microsys Srl. Microsoft MVP Azure – Application Integration Microsoft P-Seller competenza Data Platform SQL / BI / Integration Architect.

Agenda Introduzione Upload dataset su SQL Database Event Hubs Stream Analytics Upload file su SQL Database Service Bus Logic Apps

Introduzione

Introduzione Strumenti a disposizione SQL Server Integration Services BCP Bulk Copy API … Qual’è il focus di questa presentazione ? - “Consideriamo soluzioni alternative”

Introduzione Quali sono gli scenari ? IOT Application Integration B2B Integration … In generale piccoli pacchetti di dati ma frequenti

Introduzione Quali sono le sfide ? Upload dataset di sorgenti dati eterogenee e distribuite geograficamente Upload di flat file con schemi complessi

Upload datasets su SQL Database

Layers On-Prem Custom Code Az Event Hubs Az Stream Analytics Az SQL Database

Azure Event Hubs

Azure Event Hubs «Microsoft Azure Event Hubs is a managed platform service that provides a foundation for large-scale data intake in a broad variety of scenarios»

Azure Event Hubs Enterprise messaging scenarios with sophisticated capabilities sequencing dead-lettering transaction support strong delivery assurances high throughput and processing flexibility for event streams

Azure Event Hubs Partizioni Event Hub Partition 1 Partition 2 Partition N

Sviluppare un event publisher 1. Installare il NuGet Package 2. Scrivere il codice using System.Threading; using Microsoft.ServiceBus.Messaging; ... string connectionString = "{Connection String}"; string eventHubName = "{Event Hub Name}"; var eventHubClient = EventHubClient.CreateFromConnectionString(connectionString, eventHubName); eventHubClient.Send(new EventData(Encoding.UTF8.GetBytes({Data}))); // Max 256KB eventHubClient.Close(); https://azure.microsoft.com/en-us/documentation/articles/event-hubs-csharp-ephcs-getstarted/

Azure Stream Analytics «Azure Stream Analytics consente di sviluppare e distribuire rapidamente soluzioni a basso costo per ottenere informazioni approfondite in tempo reale da dispositivi, sensori, infrastruttura e applicazioni»

Azure Stream Analytics Event Consumer - Event Hub - IoT Hub - Blob Storage

Azure Stream Analytics Input Event Hub IoT Hub Blob Storage Query Output SQL Database Table Storage Service Bus * DocumentDB Power BI Data Lake Store < >

Check List Resource Group Event Hubs Namespace Event Hubs queue Stream Analytics Job Job Input Job Output Job Query

DEMO Upload Dataset su Azure SQL Database

Upload flat file su SQL Database

Introduzione

Layers On-Prem Custom Code Az Service Bus Az Logic Apps Az SQL Database

Azure Service Bus Una coda con un meccanismo di publish-and-suscribe

Azure Service Bus - Queue Comunicazione unidirezionale Ogni coda funge da intermediario (AKA broker) che archivia i messaggi inviati fino a quando non vengono ricevuti Ogni messaggio viene ricevuto da un singolo destinatario Message Publisher Queue Message Consumer

Sviluppare un event publisher 1. Installare il NuGet Package 2. Scrivere il codice Ricorda Qualcosa ? using Microsoft.ServiceBus.Messaging; ... string connectionString = "{Connection String}"; string queueName = "{Queue Name}"; var queueClient = QueueClient.CreateFromConnectionString(connectionString, queueName); // Max 256KB Standard Subscription – Max 1MB Premium Subscription queueClient.Send(new BrokeredMessage({data})); queueClient.Close(); https://azure.microsoft.com/en-us/documentation/articles/service-bus-dotnet-get-started-with-queues/

Azure Logic Apps «Logic Apps provide a way to simplify and implement scalable integrations and workflows in the cloud»

Azure Logic Apps

Azure Logic Apps Concetti Workflow Trigger Actions Managed Connectors Enterprise Integration Pack

Check List Resource Group Service Bus Namespace Service Bus queue API connection to Service Bus queue API connection to SQL database Storage Account App Service Plan Function App Integration Account Upload flat file schema Upload sql procedure parameter schema Upload transformation Logic App

DEMO Upload Flat File su Azure SQL Database

Risorse https://azure.microsoft.com/en-us/services/event-hubs/ https://azure.microsoft.com/en-us/services/stream-analytics/ https://azure.microsoft.com/en-us/services/service-bus/ https://azure.microsoft.com/en-us/services/logic-apps/ https://pellitterisbiztalkblog.wordpress.com/2016/11/08/upload-dataset-on-azure-sql-database-using-azure-event-hub-and-azure-stream-analytics/ https://pellitterisbiztalkblog.wordpress.com/2016/11/14/upload-flat-file-on-azure-sql-database-using-azure-logic-app/

Q&A Domande ?

#sqlsatParma #sqlsat566 Grazie !