cloudbytestorage / touchstone Goto Github PK
View Code? Open in Web Editor NEWTesting for Infrastructure w.r.t Storage
License: Apache License 2.0
Testing for Infrastructure w.r.t Storage
License: Apache License 2.0
- sanitize, sort, filter
- [ -z $vm_id ] || vm_id_list="${vm_id_list}\n$vm_id"
- vm_id_list=$( echo -e $vm_id_list | sort -g | uniq -d )
- [ "$vm_id_list" = "" ] || \
{ f_error "Conflicting VM IDs: ${vm_id_list}. Exiting" ; exit 1 ; }
- filter, extract, regex
- vm_name="${vm_found##*/}" # Remove path, leaving VM name
- vm_id=${vm_name##*[!0-9]} # Remove name, leaving VM ID
- conditional checks
- f_info "Starting all VMs in $host_vmdir"
- if [ $( ls $host_vmdir | wc -l ) = 0 ]; then
f_error "No VMs found. Exiting"
exit 1
fi
- Searches for the line which does not start with
- # or
- single quote (โ) or
- double front slashes (//)
- grep -v "^#\|^'\|^\/\/" comments
- cut usage
- extract first 3 chars of each line from a file
- cut -c1-3 test.txt
- start from 3rd character till end of each line
- cut -c3- test.txt
- first 8 characters
- cut -c-8 test.txt
- first field of each line delimited by :
- cut -d':' -f1 /etc/passwd
- fields 1 through 4, 6 & 7
- grep "/bin/bash" /etc/passwd | cut -d':' -f1-4,6,7
- select only if line contains the delimiter
- grep "/bin/bash" /etc/passwd | cut -d'|' -s -f1
- select all except specific fields
- grep "/bin/bash" /etc/passwd | cut -d':' --complement -s -f7
- change output delimiter for display
- grep "/bin/bash" /etc/passwd | cut -d':' -s -f1,6,7 --output-delimiter='#'
- change output delimiter to newline
- grep bala /etc/passwd | cut -d':' -f1,6,7 --output-delimiter=$'\n'
- grep supports some special character classes
- [:digit:] Only the digits 0 to 9
- [:alnum:] Any alphanumeric character 0 to 9 OR A to Z or a to z.
- [:alpha:] Any alpha character A to Z or a to z.
- [:blank:] Space and TAB characters only.
- These are always used inside square brackets in the form [[:digit:]]
- grep all the process Ids of ntpd daemon process
- grep -e "ntpd\[[[:digit:]]\+\]" /var/log/messages.4
- print the matched line, along with the 3 lines after it.
- grep -A 3 -i "example" demo_text
- search for a given string in a file (case in-sensitive search)
- grep -i "the" demo_file
- search for a given string recursively in all files
- grep -r "ramesh" *
- find files using file names (case in-sensitive)
- find -iname "abc.txt"
- execute commands on files found by the find command
- find -iname "abc.txt" -exec md5sum {} \;
- login to remote host
- ssh -l root remote.dns.com
- ignore white space while comparing
- diff -w abc.txt abc2.txt
- copy all images to an external hard-drive
- ls *.jpg | xargs -n1 -i cp {} /hard-drive/dir
- search all imgs in the system and archive
- find / -name *.jpg - type f -print | xargs tar -cvzf images.tar.gz
- download all the URLs frm a file
- cat url-file.txt | xargs wget -c
- file size in human readable format
- ls -lh
- order files based on last modified time
- ls -ltr
- visual classification of file with special characters
- ls -F
- view the contents of *.zip without unzipping
- unzip -l abc.zip
- check status of all services
- service --status-all
- view current running processes
- ps -ef | more
- view current running processes in a tree structure
- ps -efH | more
- view memory including the total line
- free -t
- view memory in GB
- free -g
- file system disk space usage in bytes
- df -k
- file system disk space usage in human readable
- df -h
- display the type of file system e.g. ext4
- df -T
- find where a specific unix command exists
- whereis ls
- whereis top
deprecated to [all] use cases
ps -auxfwww
moved to smartinfra
- [openstack-dev] Announcing the OpenStack Health Dashboard
- libUCL / UCL as FreeBSD config question
- https://github.com/Inuits/monitoring-plugins
- https://gist.github.com/pconerly/5498503
- http://stromberg.dnsalias.org/~strombrg/NFS-troubleshooting-2.html
- https://github.com/michaeldexter/vmrc
- https://github.com/zfsonlinux/zfs-auto-snapshot/blob/master/src/zfs-auto-snapshot.sh
- [openstack-dev] [nova][infra][bugs] Grafana Dashboard for Bugs
- https://forums.freebsd.org/threads/sharing-of-dtrace-scripts.32855/
- Eliot is a Python logging system that outputs causal chains of actions
- happening within and across process boundaries.
- A logical trace of the system's operation.
- https://github.com/ClusterHQ/eliot
discarded by references issue !!!
Each single-function-file should check its own dependencies before using them
What is a single-function-file refer - #24
iostat=`which iostat 2>/dev/null`
bc=`which bc 2>/dev/null`
# Ensuring we have the needed tools:
( [ ! -f $iostat ] || [ ! -f $bc ] ) && \
( echo "ERROR: iostat and bc needs to be installed to run this plugin\n\tuse: apt-get install iostat bc\n" && exit -1 )
reference - https://github.com/Inuits/monitoring-plugins/blob/master/check_iostat
Let us name the CLIs as below:
some samples..
2015-10-30T19:43:43.041Z cpu11:52989)FSS: 6146: Failed to open file 'A1-flat.vmdk'; Requested flags 0x4001, world: 52989 [sdrsInjector], (Existing flags 0x4008, world: 47981 [vmx]): Busy
2015-10-30T19:43:43.043Z cpu11:52989)FSS: 6146: Failed to open file 'A1-e1331c4c.vswp'; Requested flags 0x4001, world: 52989 [sdrsInjector], (Existing flags 0x8, world: 47981 [vmx]): Busy
2015-10-30T19:43:43.055Z cpu0:52989)FSS: 6146: Failed to open file 'A2-e1331c4d.vswp'; Requested flags 0x4001, world: 52989 [sdrsInjector], (Existing flags 0x8, world: 48786 [vmx]): Busy
2015-10-30T19:43:43.057Z cpu0:52989)FSS: 6146: Failed to open file 'A2-flat.vmdk'; Requested flags 0x4001, world: 52989 [sdrsInjector], (Existing flags 0x4008, world: 48786 [vmx]): Busy
2015-10-30T19:44:21.956Z cpu4:32811)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x439d8b8fca00, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
2015-10-30T23:14:49.955Z cpu12:33028)StorageApdHandlerEv: 110: Device or filesystem with identifier [2c47b2e7-7d66feab] has entered the All Paths Down state.
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aeba8330 3
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aeaff210 3
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aeb98f70 3
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aeaf4950 3
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aebd56f0 3
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aeb8ee30 3
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aebd9110 3
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aeafcc90 3
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aeb93750 3
2015-10-30T23:14:57.953Z cpu7:33081)NFSLock: 612: Stop accessing fd 0x4302aeba7610 3
2015-10-30T23:16:37.955Z cpu14:55812)WARNING: NFS: 338: Lost connection to the server 14.10.1.102 mount point /VMWAREWin-2TB, mounted as 2c47b2e7-7d66feab-0000-000000000000 ("Windows")
// vsm ip is 14.10.1.102
2015-10-30T23:16:49.953Z cpu17:33091)WARNING: NFS: 338: Lost connection to the server 14.10.1.103 mount point /VMWARELinux-2TB, mounted as 2dc277f5-36cdd4e4-0000-000000000000 ("Linux")
// vsm ip is 14.10.1.103
2015-10-30T23:23:20.306Z cpu3:63069)WARNING: NFS: 2208: Failed to get attributes (No connection)
2015-10-30T23:23:20.306Z cpu3:63069)NFS: 2264: Failed to get object 36 2c47b2e7 7d66feab dfbdd01e 25c583de 4000a 0 b8 0 0 0 0 0 :No connection
2015-10-30T23:23:43.194Z cpu3:63071)WARNING: NFS: 2208: Failed to get attributes (No connection)
2015-10-30T23:23:43.194Z cpu3:63071)NFS: 2264: Failed to get object 36 2c47b2e7 7d66feab dfbdd01e 25c583de 4000a 0 b8 0 0 0 0 0 :No connection
2015-10-30T23:42:58.182Z cpu4:34759 opID=73d59e4)FSS: 5327: No FS driver claimed device 'mpx.vmhba32:C0:T0:L0': No filesystem on the device
2015-10-30T23:46:42.113Z cpu3:63688)WARNING: NFS: 2208: Failed to get attributes (No connection)
2015-10-30T23:46:42.113Z cpu3:63688)NFS: 2264: Failed to get object 36 2dc277f5 36cdd4e4 a4834153 71f390de 4000a 0 c5 0 0 0 0 0 :No connection
2015-10-30T23:49:21.961Z cpu15:32822)NMP: nmp_ThrottleLogForDevice:3178: Cmd 0x1a (0x439d8bb76980, 0) to dev "mpx.vmhba32:C0:T0:L0" on path "vmhba32:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x24 0x0. Act:NONE
@AbhiCloudbyte Please structure the project into specs, bugs, scripts etc.
The specs in turn can have below:
env.md
run.md
bug-reproduce.md
apply-patch.md
-- apply-patch-dr.md
-- apply-patch-iscsi-provisioning.md
report.md
bug-verify.md
The bugs can have shell scripts with the name equal to bug id. e.g. 7445.sh means a bug with id as 7445. It can have a function that returns a dictionary. This dictionary will be key:value pairs e.g.
desc: description of the bug
found_at: version of software
fixed_at: version of software
The scripts folder can have sub-folders e.g.
env/
run/
bug-reproduce/
apply-patch/
report/
bug-verify/
Possible Steps:
Some devd.log samples
Fri Oct 30 20:45:35 2015 : !system=GEOM_MULTIPATH subsystem=C1R6-gRq6 type=ERETRIES_EXHAUSTED
Fri Oct 30 20:45:35 2015 : !system=GEOM_MULTIPATH subsystem=C3R5-gRq6 type=ERETRIES_EXHAUSTED
Fri Oct 30 20:45:35 2015 : !system=GEOM_MULTIPATH subsystem=C4R5-gRq6 type=ERETRIES_EXHAUSTED
Fri Oct 30 20:45:47 2015 : !system=GEOM_MULTIPATH subsystem=C2R6-gRq6 type=ERETRIES_EXHAUSTED
Fri Oct 30 20:45:47 2015 : !system=GEOM_MULTIPATH subsystem=C3R5-gRq6 type=ERETRIES_EXHAUSTED
Fri Oct 30 20:45:47 2015 : !system=GEOM_MULTIPATH subsystem=C4R5-gRq6 type=ERETRIES_EXHAUSTED
...
Fri Oct 30 21:36:33 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=register
Fri Oct 30 21:36:33 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=unregister
Fri Oct 30 21:36:33 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=register
Fri Oct 30 21:36:33 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=unregister
Fri Oct 30 21:37:08 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=register
Fri Oct 30 21:37:08 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=unregister
Fri Oct 30 21:37:08 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=register
Fri Oct 30 21:37:08 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=unregister
...
Fri Oct 30 19:29:11 2015 : !system=ZFS subsystem=ZFS type=ereport.fs.zfs.io class=ereport.fs.zfs.io ena=475374330874695681 pool=Pool pool_guid=4239232734146889058 pool_context=0 pool_failmode=continue vdev_guid=1630973112376327302 vdev_type=disk vdev_path=/dev/multipath/C2R6-gRq6 parent_guid=6316210389471247725 parent_type=raidz zio_err=5 zio_offset=29468317184 zio_size=8192 zio_objset=68 zio_object=1602 zio_level=0 zio_blkid=134864
...
[{rabbit, [{vm_memory_high_watermark,3.090625},{log_levels,[{connection,warn}]}]}].
workbench verify failure @ switch
[Nov 13 2015 10:04:56 : DEFAULTAUT] [0] WARN: Failed to connect to the Switch,
Exception: Failed Connect to the Switch,
Exception: read error: No child processes. /usr/bin/ssh:
symbol lookup error: /usr/bin/ssh: undefined symbol: FIPS_mode
[Nov 13 2015 10:04:56 : DEFAULTAUT] [0] WARN: SSH and Telnet are currently
supported only for Physical Switches
workbench verify failure @ vcenter
[Nov 13 2015 09:52:49 : STAFBASE ] [0] FRAMEWORK: Executing STAF command:
staf localhost VTAF_Host connect agent 14.10.1.13 userid vsphere.local@administrator
password :9:Test@1234 ssl
[Nov 13 2015 09:52:53 : STAFBASE ] [0] FRAMEWORK: RC:5169
[Nov 13 2015 09:52:53 : STAFSDK ] [0] WARN: Error on Connection to 14.10.1.13, RC:5169,
...:Cannot complete login due to an incorrect user name or password.:9:...
[Nov 13 2015 09:52:53 : MAIN ] [0] ERROR: Trouble creating test instance,
Error on Connection to host 14.10.1.13
workbench staf install failure
something related to "Invalid Login"
while trying to add ESX at Workbench's Installation Helper window:
add a device called /dev/sdc
disk /dev/sda: 21.4 GB, ...
disk /dev/sdb: 3221 MB, ...
disk /dev/sdc: 3221 MB,...
echo "scsi add-single-device <H> <B> <T> <L>" > /proc/scsi/scsi
<H> : Host
<B> : Bus (Channel)
<T> : Target (Id)
<L> : LUN numbers
echo "scsi add-single-device 0 0 2 0">/proc/scsi/scsi
fdisk -l
disable ipv6 settings in Windows
enable auto logon in Windows
ESX does not have VMware paravirtual SCSI controller type VMs.
In one Windows VM e.g. 64 bit 2008 r2
set the scsi controller to para virtual
while installing go for custom install
the driver wont be detected
from the floppy drive of esx select the paravirtual driver of the windows
rescan
format the primary partition
Lets check if fabric can be used to upload & download files.
download for backup purposes
download for parsing purposes
upload for update purposes
upload for restore purposes
It would be ideal to download the original conf from existing environment into the test runner environment. Then parse the conf to get below details:
Fri Oct 30 21:49:28 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=register
Fri Oct 30 21:49:28 2015 : !system=NFS_VOLUME subsystem=14.10.1.11,1 type=unregister
//Here, 14.10.1.11 is the gateway.
This enables reboot etc without prompts. Vital for automation scripts.
test with partial upgrade
- upgrade elastistor but not elasticenter
- test various features
- check if supportability criteria is met
- upgrade elasticenter but not elastistor
- test various features
- check if supportability criteria is met
io meter, disaster recovery & clone
- setup
- run various EC list commands
- compare the time taken with some preconfigured time
- run
- create iscsi volume
- mount @ client side & run IO Meter
- create DR on another site
- run overnight
- create a clone for the iscsi volume
- analyze
- same as setup
- check the IO Meter status
- run zfs commands over ssh
- stop IO Meter & repeat analyze steps
eccli & error injection
- build error injection into response of eccli
- this will help in testing clients s.a openstack
eccli & multi type responses
- response in cli format
- response in json format
escli cbdpd health
- escli vsm list
- escli cbdpd run status --vsmid
- escli cbdpd lsof --vsmid
- escli cbdpd job list --vsmid
- escli cbdpd job status --vsmid --jobid
- escli cbdpd job log status --since x-minutes --vsmid --jobid
- escli cbdpd job retention check --vsmid --jobid --retention
- NOTE - We shall remove the snapshot comparison to a different escli call.
best practices
- use-case should have an estimated timeout property.
- steps within a use-case should have a timeout property.
deprecated to [all] commands, troubleshooting, monitoring, logging
deprecated to [all] use cases
Write a spec as well script:
DB in Sync
Node Uptime (Days)
Free Memory Alert
Swap Used
OS Disk Free
tpool Disk Free
CPU
Mgmt Network Up
Data Network Up
Disk Status
CDB
Volume Disconnects
Rabbit MQ Connections
Throttling
Retransmission Packets %
Duplicate Packets %
An overview of the folder structure:
sector size
Disk /dev/sdm: 1073.7 GB, 1073741824000 bytes
255 heads, 63 sectors/track, 130541 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes (wrong size)
I/O size (minimum/optimal): 32768 bytes / 1048576 bytes
Disk identifier: 0x9c046880
Disk /dev/sdk: 107.4 GB, 107374182400 bytes
255 heads, 63 sectors/track, 13054 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes (correct size )
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x0c2096b6
using procstat
- run ps command
- e.g. cat command is stuck
- # ps auxwww | grep cat
- # procstat -kk <pid that you get from ps>
- # procstat -t <pid that you get from ps>
- # procstat -S <pid that you get from ps>
- # procstat -H <pid that you get from ps>
- # procstat -f <pid that you get from ps>
Check pool, vsm & volumes to be online
zpool status
netstal -nal
Check incoming IOPs
check_iops.sh
Check available space @ EC
df -kh
check the space taken by log files
check if logs have a separate partition
Check RabbitMQ if it has lot of open/close connections
restart rabbitmq
check if rabbitmq log is getting updated else restart
Check the log levels of Elasticenter
log4j-cloud.xml
Check if EC is able to communicate to ES
via global settings
Check if /var/log/messages are being updated ?
restart syslogd
Check for sonewconn errors @ /var/log/messages
Analyze the queue depth of service
If listen queue overflow parameter is 8 then 8 = 5 * 1.5 ~ 7.5
Check for the TCP services with queue depth = 5 (from above calculations)
Check gmond & gmetad services
analyze NFS via wireshark
- tcpdump -s 0 -w fifo.pcap host <nfs-server>
- at nfs client run the operations e.g. ls -l
Is a process stuck ?
- ps axHl
- check for WCHAN
- check if it is stuck within ZFS
- TODO show an example !!!
is it NFS issue or net drivers ?
- check TSO settings for NFS
- check above on both NFS server & client
- check rsize=32768, wsize=32768 mount options for NFS mount.
cp process is slow
- if underlying storage is NFS, then check TSO
- if underlying storage is NFS, then check rsize & wsize
- check netstat -m # look for mbuf cluster exhaustion
- get the appropriate sysctl to check stats of your net driver
SSD Lifetime
- dual use of SSD for ZIL & L2
- i.e. SSD gets hammered with writes from Write Caching & L2 feeding
vfs.zfs.vdev.bio_flush_disable=1
bio_flush_disable=0
- Storage will wait for IO data to be written to the cache and actual physical location
bio_flush_disable=1
- Storage will wait for IO to be written to cache only.
- Makes response to client faster.
/sbin/gmultipath refresh
glabel status | grep root | awk '{print $3}' |awk -F 'p2' '{print $1}'
glabel status |grep -v -w Name
gpart status |grep -v -w Name
/sbin/ifconfig bce3| grep netmask | cut -d' ' -f4
/sbin/ifconfig bce3| grep status | cut -c 10-
/usr/local/bin/ipmitool lan print |grep -v "IP Address Source" |grep "IP Address" | cut -d: -f2
/usr/local/bin/ipmitool sdr type "Power Supply"
/usr/local/bin/ipmitool sdr type "Fan"
/usr/local/bin/ipmitool sdr type "Temperature"
/usr/local/bin/ipmitool -v sdr type "Temperature"
// diskPath /dev/ses0
// diskName ses0
sg_vpd --page=ei ses0 | grep -w NV_SUP=1
camcontrol modepage ses0 -m 0x08 | grep -w "WCE: 1"
// similarly
sg_vpd --page=ei pass0 | grep -w NV_SUP=1
camcontrol modepage pass0 -m 0x08 | grep -w "WCE: 1"
camcontrol devlist |grep ses |grep scbus1 |grep -v grep
/usr/sbin/getencstat -v /dev/ses1 |grep -E 'Array Device|Device Slot' |grep -v grep
/usr/sbin/getencstat -v /dev/ses1 |grep -v grep
getencstat -v /dev/ses1 2>&1 |grep dev: |grep da0 |grep -v ada0 |grep -v grep
getencstat -v /dev/ses1 2>&1 |grep 'Element 0x1:' | grep -v grep
setobjstat /dev/ses1 0x1 0x80 0x00 0x02 0x00
/sbin/sysctl -a kern.disks
/usr/local/sbin/smartctl /dev/pass0 -H | grep "SMART Health Status"
/usr/local/sbin/smartctl -x /dev/pass0|grep -i grown
/usr/local/sbin/smartctl --scan
ctladm port -l
ctladm wwpnlist
/usr/local/cb/bin/fcmgmt.sh fakeall /cf/conf/config.xml ALL ALL 0 1
/usr/local/bin/sg_logs -p 0x05 /dev/pass2
/usr/local/cb/bin/getdiskinfo.sh da0
/usr/local/bin/python /usr/local/agent/cloudbyte/sedutils.py discover da0
// H/W issues @ elastistor
camcontrol devlist
refresh hardware @ elasticenter
move to maintainance @ elasticenter
// NFS Datastore gone down
sh /usr/local/cb/bin/services_bkptenant nfs start 1 1
sh /usr/local/cb/bin/services_bkptenant nfs stop 1 1
grep bkp * | grep nfs
grep bkp * | grep nfs | grep -v 3
vi /usr/local/cb/bin/services_bkptenant
/sbin/ifconfig bce3 alias 14.10.1.103/8 fib 1
grep "14.10.1.103" /var/log/cbc/starttsm_880fd73-23-2332-ah32.log
grep "14.10.1.103" /var/log/messages
vi starttsm_89dsgd-we2-as88-asas.log
// monitoring
gstat -f multi
zpool iostat 1
netstat -nal | grep 14.10.1.103 // vsm ip
lsof -p <pid>
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.