Monday, 11 November 2019

Descriptive Statistics for Data Analytics/Science


DS

Basically descriptive statistics tells everything about the data that is available, it doesn’t tell us anything out of that data.

So how to describe the (Numerical/Categorical) data?

- graphical representation 
- tabular representation
- summary statistics 

Type of variables in statistics-


1) Numerical or Quantitative (continuous and discrete)
2) Categorical or Qualitative (only discrete)

- nominal
- ordinal
Quantitative/Qualitative



Speaking about variables they can be single or multiple as 
We deal with data in descriptive statistics, it is important to establish relationship between this variables so as to summarize.

  • Graphical representation- Single variable


- For Categorical(Qualitative) variables:

-> Bar Graph: suited for ordinal variables 
-> Pie Chart: suited for nominal variables

- For Quantitative(Numerical) variables:

-> Box-plot( suited for continuous data)
-> Histogram(1st step to start before distribution)
-suited for discrete data


  • Graphical Representation: Multiple variables


- Scatter plots-> two quantitative (numerical) variables
- Box-plots-> one categorical (qualitative) other quantitative variable (Suited for continuous data)
- Contingency table -> both categorical with frequency of occurrence of both.

  • Measures of Central Tendency: 

How to summarize data through numbers?(summary statistics)

- Measures of central tendency (mean, mode, median)
- Dispersion
- Skew and Kurtosis
central tendency


Choosing when to use mean, median:

- bad outlier
- good outlier

When to use Mode:

- useful with nominal variables 
- multi modal distribution

  • Measures of dispersion:

- Range (Max- Min)
- InterQuartile Range (IQR) = 3rd Quartile - 1st Quartile or (75th percentile - 25th percentile) 
- Sample Standard Deviation
- Variance
- Mean Absolute Deviation
Dispersion



 All Formulas:


DS formula









Sunday, 22 September 2019

How Amazon Echo works??


Echo

The amazon echo is a smart IOT device which listens to your voice and translates it into commands to perform tasks such as Turn ON/OFF lights, playing music, ordering products from amazon, setting alarm, etc. Read this blogpost to know how it works!!!


What is Amazon Echo?

Amazon echo is a device which uses speech recognition to perform tasks, the actual brain which performs these tasks is Alexa. Alexa  uses a small computer along with microphones, speakers, and cloud computing (AWS). It performs a wide range of tasks using the 'Internet'. All you need is smart devices around you to make use of the echo fully. 

How Echo/Alexa works?

Echo is always ON, it connect's to the home wifi and stays ON. When you say something "Alexa..." it wakes up, Alexa records that voice and internally gathers voice commands and send them to the natural langauge voice recognition service in the clouds called Alexa Voice Service it interprets the commands and sends back the result.

Check this graphical representation to know how it works.

echo

 What are Alexa compatible devices?
  • Echo
  • Echo dot
  • Echo show
  • Echo spot
  • Echo plus
  • Echo Input
  • Echo Show 5
  • Echo Tap
Buy Echo devices from amazon Buy here

What all home devices are compatible with Alexa other than Echo (smart speaker)?

Arlo Q - Smart Camera buy here
Alexa

Ecobee (5th gen) - smart thermostat buy here

amazon echo







Philips hue white A19 starter kit - smart lights  buy here
Alexa




















Check this post for more updates about amazon Alexa. 

Saturday, 31 August 2019

Delete the Cam scanner app now!!!

camscanner malware

Your portable mobile scanner is infected with malware!!

We come across many apps in our smart phones which triggers lots of advertisements(almost all free apps). As we tend to use free version of apps, we don’t have privilege to disable the Ads. If you want seamless ad free experience you need to spend bucks for purchasing the premium version of app. Recently cam scanner was surrounded with new controversy which looks very scary‼️‼️

Cam Scanner has been removed from the plays store!!

The malware is Trojan Dropper, it’s a software which runs and downloads additional modules on the smartphone without the user consent. In case of cam-scanner the malware runs multiple ads on the phone  and make user signup for paid services without there knowledge. Now let’s understand in layman terms how critical is this issue!!!

What is Malware?


malware trojan


Malware is a program/file that is harmful to a computer/smartphone user. Malware can be viruses, worms, Trojan horses or spyware. This program performs different types of functions like stealing user data, taking control over a specific computer function or system. Malware can monitor or even delete sensitive data without user's permission.

What is Cam scanner?


It's a widely used mobile app which converts your phone into a portable scanner. It helps to scan documents, ID's, images etc. It automatically crops the edges and enhance the scanned document. It has more than 100 million downloads on play store.

Why should I uninstall the Cam scanner app(Android users)?

delete camscanner

Prior to uninstalling the app you should first take backup of your important scanned data in the app. 

Steps To take backup of your files from Camscanner:


Till now the app was running smoothly but in recent versions of the app the security researchers have found malicious malware which was contained in app's advertising library, this malware is known as Trojan-Dropper.AndroidOS.Necro.n. So what this Trojan malicious code does is downloads more malicious codes into your smartphone and make you sign up into paid services without your knowledge!!

You should delete the app from your phone until the app developers officially announce and upload a stable malware free version on the play store.

  • There are many old versions of the app on play store like Cam Scanner HD etc. but you should avoid downloading such versions as it may contain the malicious code.
  • The ios users don't need to worry about this as due to apple's strict privacy policy the app won't affect the apple users and it's malware free.

What if i need the stable malware free version ?


If you still need malware free version now, download the apk file provided by camscanner. 
Download the malware free apk file now by clicking this link- https://t.co/79tt0IrMdw

Official Statement from Camscanner:


Steps To take backup of your files from Camscanner:



All users should be cautious about the usage of app if you have scanned your sensitive documents please move them to different location than delete the app, your data is your identity.

Thursday, 22 August 2019

FaceApp : Good or Bad!!

Everything you need to know!!!!!

cover image



Almost everyone who has a smartphone must have heard of FaceApp. Basically it's an App which makes you look old. This App first went viral in 2017 but now in 2019 it came into limelight it has become viral sensation overnight. Almost every person has use this app or must have come across his/her friend or relative's transformed picture. The app has been downloaded millions times in just a couple of days.
Even many celebritries posted their pictures on instagram and twitter and used the hashtag #FaceAppChallenge and this way the app gained emense popularity.


faceapp google search


The real question is how this app performs so well and how it gives such accurate results? You'll get to know the answer in the end ;)

What is FaceApp?



FaceApp is a free mobile photo editor app, it makes a person look older/younger with just a click of a button. You can download the app from Google Play Store or Apple App Store. There are some premium features in the app for which you need to pay, but you can look old for free!!



How to use FaceApp?

What you see on the app is just a simple UI consisting of :
Click a selfie or choose a photo from your phone's gallery.

How FaceApp works?


  • In layman terms when you click a selfie or choose it from gallery Faceapp upload that photo in it's server and uses the Facial algorithm to process the filters you select.
  • The app is being trained with millions of photos of old age people.It uses server side technology to function. In more technical terms it does nueral tranformation in your face ie. smile/not smile, male/female, young/old etc.
  • In Machine Learning terms it uses deep learning to give highly accurate results.In one of the article it was mentioned that FaceApp uses "deep generative convolutional neural networks".
  • It's hard to predict the exact algorithm/technique as the plus point the makers have is they got millions of face data when the app went viral(back in 2017) and eventually they trained the algorithm with these data to give more precise results about you being old.

It's AI that gives you those wrinkles with a pinch of filter!!


(It's the data on which the algorithm is trained on, which makes it so accurate!! (Answer to the question )


Do I need to worry about my data ?


Unlike other photo editor apps FaceApp doesn't run locally on your phone ie. it uploads your pic of it's server their it uses all those aging/deaging filters and than it gives you your older version.So basically you yourself are allowing the app to use your image.


GOT meme of faceapp


One of the privacy policy of FaceApp says: (They've removed this 'User contect' privacy policy from their site!!!)

privacy policy
Privacy Policy Source: https://www.faceapp.com/privacy-en.html

How Can I delete my pic?

The app maker say they delete almost all pics (not all) so what they do with other pics??
If a user want's his picture to get deleted. He needs to send a delete request via the app using below steps:
  1. Settings
  2.  Support 
  3. Report a bug 
It's important that we take a note of their privacy policy which says:

"FaceApp may share User Content and your information (including but not limited to, information from cookies, log files, device identifiers, location data, and usage data) with businesses that are legally part of the same group of companies that FaceApp is part of, or that become part of that group (Affiliates)"

Though they have written it "may" share the data, who know's what they do with the users data?? 
Training the algorithm is one of the objective for enhancing app's performance but  nowadays the face data can be used for illegal part as well. There are technologies which takes certain data points from your face and that help to create the faceid dummy and it'll help to gain access where their is facial recognition system.



It depends on you as a user weather you want to get old happily or you want to age using an app filter ;) 



   

Sunday, 18 August 2019

Best Data Analytics Documentary you should watch now!!

A netflix documentary on Data Scam


"Data is the new Oil ? NO: Data is the new soil"
-David Mccandless

Do you know how much data we create every single day?

- It's over 2.5 quintillion bytes of data and it's continuosly growing at an unimaginable speed.


Last week I watched this eye-opener documentary on Netflix - 'THE GREAT HACK'. It's a must watch data analytics documentary on how a company used data to create bias among the people and change their perception using data analytics.Being into the field of data i feel it's very important that everyone should be aware of the consequences of how your data can be misused if you knowingly/unknowingly share your personal information publically.


This documentary shows how a company used more than 5000 data-points(In analytical context any single piece of information is a data point) to hack peoples mind in the US election, so the person will vote for the candidate he didn't wanted to vote and the company achieved this by the buzz word we all have heard of - Data Analytics .Sounds creepy right!!




data analytics meme

In a broader sense it's like a psychological warfare where you persuade people by using communication as a means and that too using their own data with their own consent! The documentary shows how the company used the data points of people from the social media datasets and created content by doing Analytics on that data than they used it to persuade them ie. used it for biased politics in the US election.

Let me explain in layman terms how it works!!  

You take the data from a person via social media feed, survey etc.  and  using data analytics you connect all the data points for this person like (likes, dislikes, interests, etc.),than you generate the content which are displayed in the form of ads and you continuosly trigger these ads in that particular person's feed and lastly this changes the perception of the person and he/she is not even aware that they are the victim here. And this is how you persuade the people to achieve the goals via technology such as machine learning, data analytics, data science. 


data US elections



Things you should remember when it comes to data:


  • It's high time we need to think twice before we click on any suspicious link. 
  • Never post any of your confidential data on any social media platform. 
  • Don't fill any survey form before checking it's authenticity (you'll be giving your data yourself). 
  • Stop downloading any app that asks for irrelevant permissions. 

data meme

You'll get to know more of these facts when you'll watch the documentary. About the authenticity of how true the events are! All the whistle-blowers are former employees of the accused firm like Data Scientist, Business Analyst etc. 


Will keep you updated with such Data related posts, share this if you find it interesting. Till than bye.

Documentary link - https://www.netflix.com/in/title/80117542 




data meme


Sunday, 4 June 2017

Met shakthimaan !



Firstly I heard him in Pycon Pune reciting  "Nursery rhymes " ;) . It was one of the best thing in Pycon. Got to know from a meetup friend that he wrote a book on " I want to do a project tell me what to do ?". Had the same question since the time I started with python. So I Ordered the book  and started reading it. But I never thought that I'll be attending a hand's on workshop by the author himself on the same topic, sound's different but it happened. 
The workshop started at 10 am and shakthimaan was ready to help us ;), the teaching approach which shakti uses is just awesome every single slide which he displays has meaning .He took hand's-on workshop on git , I tried  git on windows (git bash) and this time I used Fedora. Previously I referred some YouTube videos , tutorials and what not But still the logical approach was missing,  but this workshop helped me in a logical sense .It not only helped me build my logic but it helped me get all basics about "FOSS", "getting started with contribution", "mailing list guidelines", "opensource" etc. 
Previously I was doubtful about the mind-map, how to select area of interest and how to focus on it and being " jack of all trades and master of  One ", Thank you shakthimaan for helping a beginner like me.



Sunday, 28 May 2017

Python May Meetup

The meetup was scheduled at 10:30 am in RedHat , pune . I reached sharp at 10 am .Met new people and had a great interaction with them as this time the crowd was new and enthusiastic.
The 1st was a lightning talk by Poruri Sai Rahul on  "Creating your Personal Github pages" , it was a hand'son and very useful . Github can be used to make static pages and the great thing is we don't even need to code from scratch ,we can just fork the repo's on github and using Bootstrap we can make static pages , there are lots of cool themes one can use - http://getbootstrap.com/getting-started/#examples , no need to write all those HTML pages from scratch containing "tags/head/body/..bla bla" just fork any basic HTML repo and make changes whatever you want.
After the Break there was a talk by Krishnakant Mane on Emacs for development , such an informative talk it was followed by jokes :) . He explained most of the things of Emacs , and being a newbie i got to know new things  . As previously i was aware of the Programming environments (Pycharm) but Emacs is like killer of all , He explained the stuff in a very interactive way and it was great listening to him .He is very experienced and knowledgeable person.
The Emacs was followed by Creating RESTfull API using python Pyramids , this was hand's on workshop about the Pyramid Framework . I was totally unaware of this thing but it is very powerful and efficient Web Framework .This workshop was conducted by Krishnakant Mane again and Prajakta. They covered most of the parts of Pyramids and it was 2 hours long but very informative .

Overall in May python Meetup got to learn lots of new Python Libraries and Frameworks , will be implementing those in future :) .



Sunday, 19 February 2017

Attended pycon pune 2017

Came for the language , stayed for the community .
-  Brett Cannon
As the Pycon was top item in my Bucket list finally the wait was over as pycon Pune chapter was on, as the venue for the conference changed it was now "Amanora The Fern Hotels and Resorts" . It was 1 hour drive from my place , I woke up at 7 and zhoooom I reached before time .
I got my badge and acquired the middle row seat, the conference started with Honza Král as the keynote speaker for pycon. As I was surrounded by developers I thought this might not be my piece of cake as I am new to python and so far I am still learning stuff, but Honza's keynote talk inspired me a lot and the fear of being a beginner was gone , he explained the elastic search thing and it did hit my mind as it was completely new concept I got to know about. 
In the tea break I met Honza, I had a conversation with him and I asked him lots of queries, he guided me in the best possible way, and I became fan of the Django developer.
Other talks were also good, the one for Documentation was very impressive , GNU Mailman 3 talk by Florian Fuchs was a great, I had a chat with him in lunch break he is superb mentor and a great pythonista.
John hawley "pearl guy" was the star of pycon  , he deals with micro python and he has a great sense of humour :D.
I met python Pune community members and I am there fan , as each one of them is very down to earth and very helpful. I met rahul Bhaiya , suraj Bhaiya , chandan Bhaiya , kushal da and other community members . The pycon was moreover organised by python Pune group. I did met python Hyderabad members and they were great. 
I got different logo stickers from Redhat, Elastic-search, reserved-bit , python etc, the pycon t-shirt is one of the coolest Tee in my collection ;).
Each volunteer in Pycon Pune worked day and night to make this conference a grand success, I saw their co-ordination and everyone was .
The developer's met were very friendly and talented, I met a very good human being pradeep shetty he is very inspiring personality.
I feel lucky to be a part of such a big community whose members are more like family. I guess I was the first guy to book the ticket for conference ;), as the booking started in the month of November and I booked my seat the very first day;). Waiting for Pycon India  2017 Delhi chapter as I did stayed for the community and I am damn happy.
Happy coding ☺

Simple Python Programs (For concept building).

Level - Beginner

(I prefer Pycharm Community Edition 2016.2.1 and python version 2.7 for coding , Download links for both are mentioned below ).



For coding you can use Python IDLE or cmd also .


Consider this programs for easy understanding of syntax , try it on your own .

1] If/else/elif  ~
#if else

print "this is shivam hello folks!!"
age = input("please enter your age==>")

#if else with numbers
if age > 18:
    print "hello sir you are allowed in pub"
elif age == 18:
    print "dude show you're id card"
else:
    print "dude you are just an infant"
#lets do this if, else thing with strings
name = "john"
if name is "sam":
    print "hello sam"
elif name is "robert": #you can use is for matching or you can use ==    
    print "hello robert"
elif name == "john":
    
    print "hello john , welcome to Mugshot Lounge"
else :
    
    print "he didnt came !!"

#output
"""C:\Python27\python.exe C:/Users/Shivam/PycharmProjects/untitled1/ex1(if).py

this is shivam hello folks!!
please enter your age==>25
hello sir you are allowed in pub
hello john , welcome to Mugshot Lounge
Process finished with exit code 0
"""
2] FOR loop ~


print "hello guys in this program let's learn about for loop"
foods = ["chinease","thai","indian","desert"] #list
print foods[0]

print foods[1]

print foods[2]

print "printing one by one the elements of the list is hectic job"  

#lets do it in a simple way

for f in foods: #lets not use i always ;)    
    print f  #this will print all list elements
print "\n"
#more with the for loop
for f1 in foods[:2]: #stop at 2nd position (slicing)    
    print f1
    
    print len(f1) #the "len" function will print the length of each element of list

#output
"""C:\Python27\python.exe C:/Users/Shivam/PycharmProjects/untitled1/ex2(for).py
hello guys in this program let's learn about for loop
chinease
thai
indian
printing one by one the elements of the list is hectic job ,


chinease
thai
indian
desert

chinease
8
thai
4
Process finished with exit code 0
"""
3] RANGE function ~
#lets learn range function

for i in range(10): #lets use i this time    
    print i

print "\n"
#more thingss
for x in range(5):
    
    print "shivam is awesome"

print "\n"
for x1 in range(5,15): #including 5 excluding 15 ie. 5,6,7,...,14    
    print x1


#output
"""
C:\Python27\python.exe C:/Users/Shivam/PycharmProjects/untitled1/ex3(range).py
0
1
2
3

4
5
6
7
8
9

shivam is awesome
shivam is awesome
shivam is awesome
shivam is awesome
shivam is awesome

5
6
7
8
9

10
11
12
13
14
"""
4] WHILE loop ~
#while loop

print "in this program lets learn about while loop"
number = 6
while number < 10:
    
    print number  #as the condition is always gonna be true
    
                  #thus the loop will be infinite so adding another statement    
    number += 1   #incrementing the number by 1

#output
"""
C:\Python27\python.exe C:/Users/Shivam/PycharmProjects/untitled1/ex4(while).py
in this program lets learn about while loop

6
7
8
9
Process finished with exit code 0

"""
5] Comments ~
print "comments are very useful in documentation"

#this is a single line comment
#this is also a comment
"""documentation interlude
or
multiple line comment"""
print "shivam" + " is a good guy" #concatinating two strings
print  'shivam',9 #concatinating string nd number , use comma
#output
"""
C:\Python27\python.exe C:/Users/Shivam/PycharmProjects/untitled1/comments.py

comments are very useful in documentation
shivam is a good guy
shivam 9
Process finished with exit code 0
"""
6] Continue ~
numberstaken = [2, 5, 9, 16, 17]

# when continue will execute nothing will be printed after it , 
#it will go to start of the loop
print "numbers tht are not taken are :"
for n in range(1, 20):
    
    if n in numberstaken:
        continue        print n

#output
"""
C:\Python27\python.exe C:/Users/Shivam/PycharmProjects/untitled1/continue.py
numbers tht are not taken are :
1
3
4
6
7
8

10
11
12
13
14
15
18
19
Process finished with exit code 0
"""
7] Break ~
number = 24

for n in range(101): #including 100       
    if n is number:
   
        print n,"yess! we found our number"           
        break #break out of loop or stop executing       
    else:
   
        print n


#output
"""C:\Python27\python.exe C:/Users/Shivam/PycharmProjects/untitled1/continue_n_break.py
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

16

17
18
19
20
21
22
23
24 yess! we found our number
Process finished with exit code 0
"""
8] Functions ~
def hello():

    print "hi guyss this is a function"
def area_of_circle(radius):
    
    area = radius * 3.14        print area

hello()

area_of_circle(25)

#output
"""
C:\Python27\python.exe C:/Users/Shivam/PycharmProjects/untitled1/functions.py
hi guyss this is a function
78.5
Process finished with exit code 0
"""

Best Data Analytics Documentary you should watch now!!

"Data is the new Oil ? NO: Data is the new soil" -David Mccandless Do you know how much data we create every single...