Posts tagged git

git: split large repo into several smaller repositories

posted on 2016-03-15 17:23:56

Here splitting a single bigger repository into several repositories will be described, preserving to each its relevant history.

This guide assumes you have:

  • A running git server. (Here gitolite was used.)
  • Sufficient system administration knowledge so you won't fuck up your companies version control.

create new repositories

First step is creating the repositores which will later be used after the split. Backup the fresh repositories up is a rather smart idea in case you ned to restart from scratch, so you do not have to recreate them through gitolite.

clone the initial repository

git clone --no-hardlinks <src_repo> <name_of_new_smaller_repo>

METHOD 1: extract the relevant subrepo data

cd <name_of_new_smaller_repo>
git filter-branch --subdirectory-filter <path_to_relevant_subfolder> HEAD

METHOD 2: remove folders but keep the rest

This is untestet, but kept here for documentation purposes.

git filter-branch --tree-filter "rm -rf <name_of_folder_to_be_removed>" --prune-empty HEAD

remove old unneeded refs and other unneeded git metadata

git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git reflog expire --expire=now --all
git reset --hard
git gc

set the origin of your newly create subrepo

... so it will use the remote repo created in the first step.

git remote rm origin
git remote add origin <your_git_user>@<your_git_server>/<reponame>
git push origin master

git submodules

posted on 2015-05-22 01:12:29

git submodules are the solution to having a repository referencing another repository.

A use case for me is using my emacs dotfile repo from within the main dotfile repository.

## clone with submodules
git clone --recursive <repo>

## after clone, but without '--recursive' initialize submodules
git submodule update --init

## status
git submodule status

## add submodule, dont forget to commit the change afterwards
git submodule add <repo-url> <reponame>

## update submodule status
git submodule update

That should be submodules in a nutshell.

github: create/delete repo via API

posted on 2015-01-30 13:46:21

This work on github's API in version 3.

Create

curl -u <user> https://api.github.com/user/repos -d '{ "name": "<reponame>", "description": "<description>" }'

user, reponame, description are to be set accordingly. description is optional.

If a user:pw combo were given, you'd not be prompted for you password. Only downside is, you have it in your bash history. (Of course, depending on your shell settings, if the command is prepended with a ' ' (Space character), this might not neccessarily be the case.)

Delete

curl -u <user> -X DELETE https://api.github.com/repos/<user>/<reponame>

Pretty self-describing.

Examples

First a creation:

[jl@jerrylee ~]% curl -u sjas https://api.github.com/user/repos -d '{"name": "my_testrepo", "description": "this is a description"}' 
Enter host password for user 'sjas':
{
  "id": 30072166,
  "name": "my_testrepo",
  "full_name": "sjas/my_testrepo",
  "owner": {
    "login": "sjas",
     ...

     ...
[jl@jerrylee ~]%     

And a deletion:

[jl@jerrylee ~]% curl -u sjas -X DELETE https://api.github.com/repos/sjas/my_testrepo
Enter host password for user 'sjas':
[jl@jerrylee ~]%     

If the deletion was successful, no response is provided. If it failed, github will tell you.

Fix git mergeconflicts (with vim)

posted on 2014-09-06 13:11:07

If, during a pull (or rebase), git fails to finish its currently run command, this one-liner might be helpful:

vim `git diff --diff-filter=U --name-only`

It will open all files, that git could not merge, in vim.

Fix the first one, save, :n (next window), fix the next one, and so forth.

Once you are done, git add . them all, git commit, and start again where git left off. (The rebase or push or whatever.)

git remote tag handling

posted on 2014-07-30 17:59:50

Using git tag is easy and good clean fun.

Except when you realize tags have to be handled extra when working with them remotely.

After having created tags locally, to push them to the server:

## all tags
$ git push --tags 

## just <mytag> tag
$ git push --tags mytag

To delete tags:

## only locally, git fetch/pull will get them again!
## a single tag
$ git tag -d <tagname>

## all local tags
## getting all tags again is just a simple push.
## useful if you've tagged a lot and want a clean slate again
$ git tag -l | cut -d' ' -f1 | xargs git tag -d

## remotely
$ git push --delete origin <tagname>

## delete ALL remote tags, BEWARE!
git fetch && git tag -l | cut -d' ' -f1 | xargs -n 1 git push --delete origin

This behaviour of tag ain't overly useful when thinking in terms of subversion. But this way dev's can create all the tags they want/like without polluting the main repo.

git ping / git ls-remote

posted on 2014-07-30 17:58:24

To check, wether a remote git repository exists, use:

git ls-remote <user>@<server>:<repository>

Compiling git from source on debian

posted on 2014-06-02 19:58:58

Compiling git from source may not be the what first comes into your mind when using git. But when working with large repositories you should consider this, especially with the profiling option. The speed improvement I got was pretty impressive. (Ok this may also be related to the upgrade from 1.7.x to 2.0.0.)

Anyway, do this for a full, profiled (= speed optimized) build: ('$' here indicates running the command as a regular user, '#' indicates root.)

$ git clone --progress -v https://github.com/git/git
$ cd git
$ make configure
$ ./configure --prefix=/usr
$ make prefix=/usr PROFILE=BUILD all
# make prefix=/usr PROFILE=BUILD install

Afterwards check git --version output if you are running the correct version and be amazed by the new speedyness.

git: amount of commits per author

posted on 2014-01-21 18:21:49

To show the amount of individual commits per author, use

git shortlog -ns

while being in the repository folder.

git: show unpushed commits

posted on 2014-01-20 06:24:05

Using

git cherry -v

you can show the unpushed commits' SHA plus their messages. Without the -v flag just the SHA is shown.

Filtering git logs

posted on 2014-01-13 18:46:22

When working with git, sometimes you have to search the logs for a particular piece of information. Of course, grep can be put to use externally. But using git log's own built-in is by far a better solution.

git log | grep -i <search pattern> | tac

will result in only matched lines. tac is like cat, but returns the results in reverse order.

Mostly you would want

git log -i --reverse --grep=<search pattern>

because this will return full commits, not just matched lines!

"git commit <commitmessage>" without quotation marks

posted on 2013-12-09 23:05:46

Using git exactly how you want it to, is best done from within a shell. (At least in my former experiences, maybe the git clients improved vastly by now?)

EGit (the eclipse plugin), SmartGit, the Github clients all did not satisfy me. Eiter the clients lacked functions (which were needed and I was back to commandline anyway) or the functions did not behave as expected: Changing files permissions, but nothing otherwise, failing merges that were completely doable in vanilla git, ...) I have had my share, especially with EGit.

But typing 'git commit -am "this is my commitmessage"' gets old over time, too.

A way to fix this is to put this in your .bashrc:

function gc { git commit -am "$*" }

If you use another shell, you might want to put it in whatever initialization file gets executed upon shell startup. On a Mac, you might want to remember this difference to usual linux.

This will make for a nice shortcut:

# OLD
$ git commit -am "this is a long commit message and i do not want to type the quotation marks"

# NEW (I have 'gc' as memo for 'git commit', choose as you like.)
$ gc so this works now without quotation marks

This has to be done through a bash function, with an alias definition it will not work. Of course you can define bash aliases for other git commands you use more often (You bet I did. Actually a ton if it, considering you much I use git nowadays.).

Somewhere in the bash manual it is said to be written [that you should prefer functions over aliases]. Got not linux at hand (cygwin at this moment) and no bash manual for grepping here, so you have to look it up yourself in case you do not trust my hearsay.

Run `git pull` on all folders in current folder

posted on 2013-11-12 04:07:16

To pull newest changes into each of the repositories in your current folder, try this: (All folders lie directly one level below. No recursive check.)

for i in *; do cd $PWD/"${i}" && git pull | cat && cd ..; done

This was kind of tricky to figure out. Important was $PWD else the cd will not work properly, and the cat instead of echo so the console output gets shown.

This blog covers .csv, .htaccess, .pfx, .vmx, /etc/crypttab, /etc/network/interfaces, /etc/sudoers, /proc, 10.04, 14.04, 16.04, AS, ASA, ControlPanel, DS1054Z, GPT, HWR, Hyper-V, IPSEC, KVM, LSI, LVM, LXC, MBR, MTU, MegaCli, PHP, PKI, PS1, R, RAID, S.M.A.R.T., SNMP, SSD, SSL, TLS, TRIM, VEEAM, VMware, VServer, VirtualBox, Virtuozzo, XenServer, acpi, adaptec, algorithm, ansible, apache, apache2.4, apachebench, apple, applet, arcconf, arch, architecture, areca, arping, asa, asdm, autoconf, awk, backup, bandit, bar, bash, benchmarking, binding, bitrate, blackarmor, blockdev, blowfish, bochs, bond, bonding, booknotes, bootable, bsd, btrfs, buffer, c-states, cache, caching, ccl, centos, certificate, certtool, cgdisk, cheatsheet, chrome, chroot, cisco, clamav, cli, clp, clush, cluster, cmd, coleslaw, colorscheme, common lisp, configuration management, console, container, containers, controller, cron, cryptsetup, csync2, cu, cups, cygwin, d-states, database, date, db2, dcfldd, dcim, dd, debian, debug, debugger, debugging, decimal, desktop, df, dhclient, dhcp, diff, dig, display manager, dm-crypt, dmesg, dmidecode, dns, docker, dos, drivers, dtrace, dtrace4linux, du, dynamictracing, e2fsck, eBPF, ebook, efi, egrep, emacs, encoding, env, error, ess, esx, esxcli, esxi, ethtool, evil, expect, exportfs, factory reset, factory_reset, factoryreset, fail2ban, fakeroot, fbsd, fdisk, fedora, file, files, filesystem, find, fio, firewall, firmware, fish, flashrom, forensics, free, freebsd, freedos, fritzbox, fsck, fstrim, ftp, ftps, g-states, gentoo, ghostscript, git, git-filter-branch, gitbucket, github, gitolite, global, gnutls, gradle, grep, grml, grub, grub2, guacamole, hardware, haskell, hdd, hdparm, hellowor, hex, hexdump, history, howto, htop, htpasswd, http, httpd, https, i3, icmp, ifenslave, iftop, iis, imagemagick, imap, imaps, init, innoDB, innodb, inodes, intel, ioncube, ios, iostat, ip, iperf, iphone, ipmi, ipmitool, iproute2, ipsec, iptables, ipv6, irc, irssi, iw, iwconfig, iwlist, iwlwifi, jailbreak, jails, java, javascript, javaws, js, juniper, junit, kali, kde, kemp, kernel, keyremap, kill, kpartx, krypton, lacp, lamp, languages, ldap, ldapsearch, less, leviathan, liero, lightning, links, linux, linuxin3months, lisp, list, livedisk, lmctfy, loadbalancing, locale, log, logrotate, looback, loopback, losetup, lsblk, lsi, lsof, lsusb, lsyncd, luks, lvextend, lvm, lvm2, lvreduce, lxc, lxde, macbook, macro, magento, mailclient, mailing, mailq, make-jpkg, manpages, markdown, mbr, mdadm, megacli, micro sd, microsoft, minicom, mkfs, mktemp, mod_pagespeed, mod_proxy, modbus, modprobe, mount, mouse, movement, mpstat, multitasking, myISAM, mysql, mysql 5.7, mysql workbench, mysqlcheck, mysqldump, nagios, nas, nat, nc, netfilter, networking, nfs, nginx, nmap, nocaps, nodejs, numberingsystem, numbers, od, onyx, opcode-cache, openVZ, openlierox, openssl, openvpn, openvswitch, openwrt, oracle linux, org-mode, os, oscilloscope, overview, parallel, parameter expansion, parted, partitioning, passwd, patch, pct, pdf, performance, pfsense, php, php7, phpmyadmin, pi, pidgin, pidstat, pins, pkill, plasma, plesk, plugin, posix, postfix, postfixadmin, postgres, postgresql, poudriere, powershell, preview, profiling, prompt, proxmox, ps, puppet, pv, pveam, pvecm, pvesm, pvresize, python, python3, qemu, qemu-img, qm, qmrestore, quicklisp, quickshare, r, racktables, raid, raspberry pi, raspberrypi, raspbian, rbpi, rdp, redhat, redirect, registry, requirements, resize2fs, rewrite, rewrites, rhel, rigol, roccat, routing, rs0485, rs232, rsync, s-states, s_client, samba, sar, sata, sbcl, scite, scp, screen, scripting, seafile, seagate, security, sed, serial, serial port, setup, sftp, sg300, shell, shopware, shortcuts, showmount, signals, slattach, slip, slow-query-log, smbclient, snmpget, snmpwalk, software RAID, software raid, softwareraid, sophos, spacemacs, spam, specification, speedport, spi, sqlite, squid, ssd, ssh, ssh-add, sshd, ssl, stats, storage, strace, stronswan, su, submodules, subzone, sudo, sudoers, sup, swaks, swap, switch, switching, synaptics, synergy, sysfs, systemd, systemtap, tar, tcpdump, tcsh, tee, telnet, terminal, terminator, testdisk, testing, throughput, tmux, todo, tomcat, top, tput, trafficshaping, ttl, tuning, tunnel, tunneling, typo3, uboot, ubuntu, ubuntu 16.04, ubuntu16.04, udev, uefi, ulimit, uname, unetbootin, unit testing, upstart, uptime, usb, usbstick, utf8, utm, utm 220, ux305, vcs, vgchange, vim, vimdiff, virtualbox, virtualization, visual studio code, vlan, vmstat, vmware, vnc, vncviewer, voltage, vpn, vsphere, vzdump, w, w701, wakeonlan, wargames, web, webdav, weechat, wget, whois, wicd, wifi, windowmanager, windows, wine, wireshark, wpa, wpa_passphrase, wpa_supplicant, x11vnc, x2x, xfce, xfreerdp, xmodem, xterm, xxd, yum, zones, zsh


Unless otherwise credited all material Creative Commons License by sjas