Jupyter notebook / Lab doesn’t open

wrote error: ‘Schema not found’

You have installed Python3 via homebrew, and you installed jupyterlab, and when you try opening jupyter notebook, you run into errors – such as command not found, or even if it opens, then you only see the directory, but not the ability to create / run a notebook.

This seems to be a know issue with brew, with Apple Chip Mac/os.

Some of the commands (after looked through couple discussion on the issue) that I ran, in the below order, that fixed the issue for me.

brew install jupyterlab
jupyter lab build
brew update
brew doctor
brew link --overwrite jupyterlab

After which, I was able to execute jupyter notebook or jupyter lab without any issues.
See if this helps you.

Visual Code – Snowflake Error: You don’t have any connection

Did you receive an error trying to execute a SQL on Snowflake from Visual Code?

Error: you don’t have any connection, even though you clearly see the extension has connection established, and you are able to see the objects, and query history in your side bar.

The reason is – that you are trying to execute a sql, that is in a .sql file extension or SQL type category in Code, instead of ‘Snowflake SQL’ type.

Make sure you change the file type in the lower right corner of the Code IDE, and you should be able to execute the query.

Hope this helps 😉

Snowflake Stored Procedure binding argument error

If you run into the below error, when trying to execute a stored procedure in Snowflake,

Execution error in store procedure

Unsupported type for binding argument

One of the causes could be, that the Execute command of the SQL is referring to a Variable that isn’t defined, most likely, misspelt.

Do check if that’s the case.

Hope this helps.

gitpython modules GitCommandError and Repo import error

You have installed gitpython using pip install gitypython in Mac.

And now you try importing the modules (from git.exc import GitCommandError or from git import Repo) that work with git repository, but then you run into the below error

>>> from git.exc import GitCommandError
Traceback (most recent call last):
File “”, line 1, in
File “/usr/local/lib/python2.7/site-packages/git/__init__.py”, line 33, in
_init_externals()
File “/usr/local/lib/python2.7/site-packages/git/__init__.py”, line 27, in _init_externals
raise ImportError(“‘gitdb’ could not be found in your PYTHONPATH”)
ImportError: ‘gitdb’ could not be found in your PYTHONPATH

If you have installed python using ‘brew install python’ or ‘brew install python@2’, then it install pip for you, as well as when you did gitpython, it would have added the gitdb. It’s seems that the folder /usr/local/sbin wasn’t in my PATH ( link ), so after I added it to my .bash_profile (PATH=$PATH:/usr/local/sbin) and ran ‘source bash_profile’, I was able to import the above modules.

Hope this helps.

Google sheet python API modules failing to install using pip

If you have tried to Install the Google Client Library, for Python in Mac, and running into error, see if the below solves the issue.

Original command documented in Google:

pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib

Error: Cannot uninstall ‘six’. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.

Modify the above command to exclude the check for already installed package ‘six’

Modified command:

pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib --ignore-installed six

Hope this helps.

Git issues after Mac Mojave OS upgrade

If you are running into Git issues / Git not being found, in Terminal or IntelliJIDEA, or any other IDEA….. see below for the solution.

Run the below commands to get the git version fixed, after Mac Mojave OS upgrade.

#1: sudo xcode-select –switch /Applications/Xcode.app
#2: sudo xcode-select –switch /Library/Developer/CommandLineTools

If you run into the error: xcode-select: error: invalid developer directory ‘/Library/Developer/CommandLineTools’
while running command #2, you would have to run the command #3, and finish the install prompt (with xcode, or just the commnadlinetools), and then run command #2 again

#3: sudo xcode-select –install

Also, note that you’ll have to run these command with ‘sudo’ previleges.

Find your Mac’ Internal IP (Local IP)

---

To find your Mac’s internal / local IP using the ‘Terminal” Application (included in Mac)

Open your ‘Terminal’ app. [ command+space, and type in ‘terminal’. it should be listed at the top]

Run the command  ifconfig | grep ‘inet’

The number highlighted (in orange) in the line with ‘broadcast’ listed in the line, is your internal IP.

SF-Senth-M:~ senth$ ifconfig | grep 'inet'
inet6 ::1 prefixlen 128
inet 127.0.0.1 netmask 0xff0000
inet6 fe0::1%lo0 prefixlen 64 scopeid 0x1
inet6 f0::9a1:a7:ec:95%en0 prefixlen 64 scopeid 0x4
inet 10.10.19.21 netmask 0xfffffe00 broadcast 10.10.169.255
inet6 fe0::84:5f:fb:df%awdl0 prefixlen 64 scopeid 0x8
inet 10.0.2.104 --> 1.1.1.1 netmask 0xfffe00

Hope this helps.

---

 

Git Config and Git Alias

Where to find git config file in Mac:

In Mac, you can usually find the Git config file in your home directory, in hidden file .gitconfig

eg., /Users/.gitconfig

How to Add alias(aliases) to Git config file:

You can add Aliases for GIT commands (a single command or a series of commands) for your convenience (easy to remember or fewer keyboard strokes)

Couple ways to add alias to the config file:

1. 1. Add using git config command: eg., here is to add alias to find the Last commit to the branch you are on.

      git config --global alias.last 'log -1 HEAD'

   2. Add the entry manually in the .gitconfig file itself. The same example can be written in the config file. Just add the below lines to the end of the config file if there is no [alias] tag in the file already, or if there is a [alias] tab already, add the alias as key value pair.


[alias]
last = log -1 HEAD


If you want the list of files that are committed, add the flag ‘–stat’.


[alias]
last = log --stat -1 HEAD


 

And the command you use to execute the alias is much like how you use other git commands.

git last

Also if you want to add an alias which takes a parameter, here is how to add git alias with a parameter.

Lets say we want to create a branch, and want to use a shorter command (‘cb’) instead of checkout.

you give an alias ‘cb’ with the actual command, wrapped in a function that takes a parameter.

[alias]
cb = "!f() { git checkout -b $1;}; f"


And you can execute ‘git cb {branch_name_you_want_to_create}‘ to create the git branch.

Feel free to provide comments/suggestions.

TroubleShoot: Table Not Found Spark-Submit

---

When you know that a table exists in Hive / Impala, either because you created it, or you are able to query the table and view it’s data through UI or CLI, and also, you are able to query the table through pyspark or spark-shell, but then when you access the same table in spark-submit command, the application throws error ‘Table not found’, then the most likely cause is that

You are using sqlContext in your SparkContext [ SqlContext(sc)] rather than using HiveContext [ HiveContext(sc) ].

Change your code from SqlContext to HiveContext, and your code would be able to find the table again !!

---

Extract Folder names from hadoop

---

If you are looking to extract all the folder names under a certain directory in hadoop, here is the command:

Say you are looking under Directory /user/{username}, the ‘hadoop fs -ls /user/{username} gives the below output’ .

drwxr-xr-x   - user  user  0 2017-08-30 13:44 /user/{username}/product
drwxr-xr-x   - user  user  0 2017-09-01 13:39 /user/{username}/quote_lines

And the below command:

hadoop fs -ls /user/{username} | grep "^d" | awk -F':[0-9]* ' '/:/{print $2}' | cut -d'/' -f5

gives the directories:

product
quote_lines

---