Python Pandas Problems Archives - Page 2 Of 10

Count the number of missing values in each column of a DataFrame

In this python pandas program, we will count the number of missing values in each column of a DataFrame.

Steps to solve the program

Import pandas library as pd.
Create a dataframe using pd.DataFrame().
Count the number of missing values in each column of a DataFrame using df.isna().sum().
Print the output.

				
					import pandas as pd
import numpy as np
df = pd.DataFrame({'Sr.no.':[1,2,3,4],
                   'Name':['Alex',np.nan,'Peter','Klaus'],
                   'Age':[30,np.nan,29,np.nan]})
print("Original Dataframe: \n",df)
print("Identify the columns which have at least one missing value:")
print(df.isna().sum())

Output :

				
					Original Dataframe: 
    Sr.no.   Name   Age
0       1   Alex  30.0
1       2    NaN   NaN
2       3  Peter  29.0
3       4  Klaus   NaN
Identify the columns which have at least one missing value:
Sr.no.    0
Name      1
Age       2
dtype: int64

identify the columns from the DataFrame which have at least one missing value

drop the rows where at least one element is missing in a DataFrame

Identify the columns from the DataFrame which have at least one missing value

In this python pandas program, we will identify the columns from the DataFrame which have at least one missing value using the pandas library.

Steps to solve the program

Import pandas library as pd.
Create a dataframe using pd.DataFrame().
Identify the columns from the DataFrame which have at least one missing value using df.isna().any().
It will print True in front of the column name if that column has any missing value or else False.
Print the output.

				
					import pandas as pd
import numpy as np
df = pd.DataFrame({'Sr.no.':[1,2,3,4],
                   'Name':['Alex','John','Peter','Klaus'],
                   'Age':[30,np.nan,29,np.nan]})
print("Original Dataframe: \n",df)
print("Identify the columns which have at least one missing value:")
print(df.isna().any())

Output :

				
					Original Dataframe: 
    Sr.no.   Name   Age
0       1   Alex  30.0
1       2   John   NaN
2       3  Peter  29.0
3       4  Klaus   NaN
Identify the columns which have at least one missing value:
Sr.no.    False
Name      False
Age        True
dtype: bool

detect missing values from a DataFrame

count the number of missing values in each column of a DataFrame

Python Pandas program to detect and display missing values from a DataFrame

In this python pandas program, we will detect and display missing values from a DataFrame using the pandas library.

Steps to solve the program

Import pandas library as pd.
Create a dataframe using pd.DataFrame().
Detect and display missing values from a DataFrame using df.isna().
It will show True if the value is missing else False.
Print the output.

				
					import pandas as pd
import numpy as np
df = pd.DataFrame({'Sr.no.':[1,2,3,4],
                   'Name':['Alex','John','Peter','Klaus'],
                   'Age':[30,np.nan,29,np.nan]})
print("Original Dataframe: \n",df)
print(df.isna())

Output :

				
					Original Dataframe: 
    Sr.no.   Name   Age
0       1   Alex  30.0
1       2   John   NaN
2       3  Peter  29.0
3       4  Klaus   NaN
   Sr.no.   Name    Age
0   False  False  False
1   False  False   True
2   False  False  False
3   False  False   True

merge two Dataframes with different columns

identify the columns from the DataFrame which have at least one missing value

Python Pandas program to merge two Dataframes with different columns

In this python pandas program, we will merge two Dataframes with different columns using the pandas library.

Steps to solve the program

Import pandas library as pd.
Create two dataframes using pd.DataFrame().
Merge two Dataframes with different columns using pd.concat([df1,df2], axis=0, ignore_index=True).
pd.concat() will merge the two dataframe but by setting the axis=0 and ignore_index=True it will merge two Dataframes with different columns.
It will show NaN if there is no record of a value in a column.
Print the output.

				
					import pandas as pd
df1 = pd.DataFrame({'Id':['S1','S2','S3'],
                   'Name':['Ketan','Yash','Abhishek'],
                   'Marks':[90,87,77]})
df2 = pd.DataFrame({'Sr.no.':[1,2,3,4],
                   'Name':['Alex','John','Peter','Klaus'],
                   'Age':[30,27,29,33]})
print('Dataframe 1: \n',df1)
print('Dataframe 2: \n',df2)
print("Merge two dataframes with different columns:")
result = pd.concat([df1,df2], axis=0, ignore_index=True)
print(result)

Output :

				
					Dataframe 1: 
    Id      Name  Marks
0  S1     Ketan     90
1  S2      Yash     87
2  S3  Abhishek     77
Dataframe 2: 
    Sr.no.   Name  Age
0       1   Alex   30
1       2   John   27
2       3  Peter   29
3       4  Klaus   33
Merge two dataframes with different columns:
    Id      Name  Marks  Sr.no.   Age
0   S1     Ketan   90.0     NaN   NaN
1   S2      Yash   87.0     NaN   NaN
2   S3  Abhishek   77.0     NaN   NaN
3  NaN      Alex    NaN     1.0  30.0
4  NaN      John    NaN     2.0  27.0
5  NaN     Peter    NaN     3.0  29.0
6  NaN     Klaus    NaN     4.0  33.0

join the two Dataframes using the common column of both Dataframes

detect missing values from a DataFrame

Join the two Dataframes using the common column and value

In this python pandas program, we will Join the two Dataframes using the common column and value using the pandas library.

Steps to solve the program

Import pandas library as pd.
Create two dataframes using pd.DataFrame().
We will perform inner join to join the two Dataframes using the common column and value using pd.merge(df1, df2, on=’Id’, how=’inner’).
We will perform an inner join on the id column.
Print the output.

				
					import pandas as pd
df1 = pd.DataFrame({'Id':['S1','S2','S3'],
                   'Name':['Ketan','Yash','Abhishek'],
                   'Marks':[90,87,77]})
df2 = pd.DataFrame({'Id':['S2','S4'],
                    'Name':['Yash','Gaurav'],
                   'Marks':[70,65]})
print('Dataframe 1: \n',df1)
print('Dataframe 2: \n',df2)
new = pd.merge(df1, df2, on='Id', how='inner')
print("Merged data:")
print(new)

Output :

				
					Dataframe 1: 
    Id      Name  Marks
0  S1     Ketan     90
1  S2      Yash     87
2  S3  Abhishek     77
Dataframe 2: 
    Id    Name  Marks
0  S2    Yash     70
1  S4  Gaurav     65
Merged data:
   Id Name_x  Marks_x Name_y  Marks_y
0  S2   Yash       87   Yash       70

join the two Dataframes along columns

merge two Dataframes with different columns

Python Pandas program to join the two Dataframes along columns

In this python pandas program, we will join the two Dataframes along columns using the pandas library.

Steps to solve the program

Import pandas library as pd.
Create two dataframes using pd.DataFrame().
Join the dataframes along the columns using pd.concat() and set axis=1.
Print the output.

				
					import pandas as pd
df1 = pd.DataFrame({'ID':[1,2,3],'Name':['Yash','Gaurav','Sanket'],
                   'Age':[30,27,28]})
df2 = pd.DataFrame({'ID':[4,3],'Name':['Tanmay','Athrva'],'Age':[26,22]})
result = pd.concat([df1,df2],axis=1)
print("New dataframe")
print(result)

Output :

				
					New dataframe
   ID    Name  Age   ID    Name   Age
0   1    Yash   30  4.0  Tanmay  26.0
1   2  Gaurav   27  3.0  Athrva  22.0
2   3  Sanket   28  NaN     NaN   NaN

join the two Dataframes along rows

join the two Dataframes using the common column of both Dataframes

Python Pandas program to join the two Dataframes along rows

In this python pandas program, we will join the two Dataframes along rows using the pandas library.

Steps to solve the program

Import pandas library as pd.
Create two dataframes using pd.DataFrame().
Join the dataframes using pd.concat().
Print the output.

				
					import pandas as pd
df1 = pd.DataFrame({'ID':[1,2,3],'Name':['Yash','Gaurav','Sanket'],
                   'Age':[30,27,28]})
df2 = pd.DataFrame({'ID':[4,3],'Name':['Tanmay','Athrva'],'Age':[26,22]})
result = pd.concat([df1,df2])
print("New dataframe")
print(result)

Output :

				
					New dataframe
   ID    Name  Age
0   1    Yash   30
1   2  Gaurav   27
2   3  Sanket   28
0   4  Tanmay   26
1   3  Athrva   22

extract only words from a column of a DataFrame

join the two Dataframes along columns

Extract only words from a column of a DataFrame

In this python pandas program, we will extract only words from a column of a DataFrame using the pandas library.

Steps to solve the program

Import pandas library as pd.
Import re library.
Create a dataframe using pd.DataFrame().
Create a function to extract only words from the record.
Extract only words from the record using re.findall(r’\b[^\d\W]+\b’, text).
It will extract words from the record and the return will give words as the output of the function.
Now apply this function to the address column of the dataframe using df[‘Address’].apply(lambda x : search_words(x)).
The lambda function will apply the created function to each row to extract words from the address and store it in the new column.
Print the output.

				
					import pandas as pd
import re
d = {'Name':['Ramesh','Suresh','Sanket'],
     'Address':['297 shukrawar peth','200 ravivar peth','090 shanivar peth']}
df = pd.DataFrame(d)
print(df)
def search_words(text):
    result = re.findall(r'\b[^\d\W]+\b', text)
    return " ".join(result)

df['words']=df['Address'].apply(lambda x : search_words(x))
print("Only words:")
print(df)

Output :

				
					0     Name             Address
0  Ramesh  297 shukrawar peth
1  Suresh    200 ravivar peth
2  Sanket   090 shanivar peth
Only words:
     Name             Address           words
0  Ramesh  297 shukrawar peth  shukrawar peth
1  Suresh    200 ravivar peth    ravivar peth
2  Sanket   090 shanivar peth   shanivar peth

extract the hash attached word from Twitter text from the specified column of a given DataFrame

join the two Dataframes along rows

Extract the hash attached word from a tweet

In this python pandas program, we will extract the hash attached word from a tweet using the pandas library.

Steps to solve the program

Import pandas library as pd.
Import re library.
Create a dataframe using pd.DataFrame().
Create a function to extract the hash attached word from the record.
Extract the hash attached word from the record using re.findall(r'(?<=#)\w+’,text).
It will extract the hash attached word from the record and the return will give word as the output of the function.
Now apply this function to the Tweets column of the dataframe using df[‘Tweets’].apply(lambda x: find_hash(x)).
The lambda function will apply the created function to each row to extract the hash attached word and store it in the new column.
Print the output.

				
					import pandas as pd
import re
d = {'Tweets':['Pune #love','#boycottmovie','enjoying #peace']}
df = pd.DataFrame(d)
print(df)
def find_hash(text):
    hword=re.findall(r'(?<=#)\w+',text)
    return " ".join(hword)
df['extracted_word']=df['Tweets'].apply(lambda x: find_hash(x))
print("Extracting#@word from dataframe columns:")
print(df)

Output :

				
					0            Tweets
0       Pune #love
1    #boycottmovie
2  enjoying #peace
Extracting#@word from dataframe columns:
            Tweets extracted_word
0       Pune #love           love
1    #boycottmovie   boycottmovie
2  enjoying #peace          peace

extract email from a specified column of a given DataFrame

extract only words from a column of a DataFrame

Extract email from a column of the DataFrame

In this python pandas program, we will extract email from a column of the DataFrame using the pandas library.

Steps to solve the program

Import pandas library as pd.
Import re library.
Create a dataframe using pd.DataFrame().
Create a function to extract email from the record.
Extract email from the record using re.findall(r'[\w\.-]+@[\w\.-]+’,str(text)).
It will extract email from the record and return will give email as the output of the function.
Now apply this function to the company name column of the dataframe using df[‘Company_mail’].apply(lambda x: find_email(x)).
The lambda function will apply the created function to each row to extract email and store it in the new column.
Print the output.

				
					import pandas as pd 
import re
d = {'Company_mail':['TCS tcs@yahoo.com','Apple apple@icloud.com','Google google@gmail.com']}
df = pd.DataFrame(d)
print(df)
def find_email(text):
    email = re.findall(r'[\w\.-]+@[\w\.-]+',str(text))
    return ",".join(email)
df['email']=df['Company_mail'].apply(lambda x: find_email(x))
print("Extracting email from dataframe columns:")
print(df)

Output :

				
					0              Company_mail
0        TCS tcs@yahoo.com
1   Apple apple@icloud.com
2  Google google@gmail.com
Extracting email from dataframe columns:
              Company_mail             email
0        TCS tcs@yahoo.com     tcs@yahoo.com
1   Apple apple@icloud.com  apple@icloud.com
2  Google google@gmail.com  google@gmail.com

get the length of the integer of a column in a DataFrame

extract the hash attached word from Twitter text from the specified column of a given DataFrame