Does anyone know bash scripting?

[sledgehammer_999]

Can anyone knowledgeable with bash scripting implement the following in a script? (possibly a function/macro whatever it is called in bash).

Let's say we have a variable named CFLAGS that contains various things. I want it parsed and have 3 new variables so that:
CFLAGS = -I/usr/local/include/libtorrent -I/usr/local/include/libtorrent -DRANDOM_DEF -DRANDOM_DEF2=456 <other stuff>
VAR1 = /usr/local/include/libtorrent /usr/local/include/libtorrent
VAR2 = RANDOM_DEF RANDOM_DEF2=456
VAR3 = <other stuff>

The order the items appear could be random. Don't assume that -I items appear before -D or at the beginning of the string. The same goes for the others.
-I is capital i, not lower case L
-I and -D must be stripped.

[sledgehammer_999]

I forgot to mention that I want to use it in another script so it must be enclosed in a function/macro/whatever it is called in bash. It will be part of the configure script in git master.

[littletree76]

Please find a Bash shell script named "parse" attached to this post. Essential section is listed below. Edit and execute the shell script to familiarise with it. Output of execution:

Wangs-iMac:~/project/qbittorrent$ ./parse
CFLAGS = "-I/usr/local/include/libtorrent -DRANDOM_DEF -I/usr/local/include/boost -DRANDOM_DEF2=456 xxx yyy zzz"
VAR1 = "/usr/local/include/libtorrent /usr/local/include/boost"
VAR2 = "RANDOM_DEF RANDOM_DEF2=456"
VAR3 = "xxx yyy zzz"

Code: Select all

### Assigment of variable to be processed.
### Note one restriction of standard Unix command line processing:
### Those arguments without command line option (eg -I/-D) must be appended at the end of command line.

CFLAGS='-I/usr/local/include/libtorrent -DRANDOM_DEF -I/usr/local/include/boost -DRANDOM_DEF2=456 xxx yyy zzz'


### Processing start here.
### Add as many command line options as necessary to option string I:D:abc.... (colon means argument required).
### Cut and paste between the two dotted comment lines.


### --------------------------------------------------------------------------------
while getopts I:D: OPT $CFLAGS
do
	case $OPT in
		I)	VAR1+=$OPTARG' '
			;;
		D)	VAR2+=$OPTARG' '
			;;
	esac
done

### Strip away extra trailing space character (actully won't harm in any Unix scripts).

VAR1=`echo $VAR1 | $sed 's/ $//'`
VAR2=`echo $VAR2 | $sed 's/ $//'`

### Those arguments without valid option (-I/-D) in command line.

VAR3=`echo $CFLAGS | $sed 's/-I[^ ]* *//g; s/-D[^ ]* *//g'`
### --------------------------------------------------------------------------------

[sledgehammer_999]

Thank. I'll try to test it later today.
Noob question: If you add a default/catch-all case in the case structure wouldn't it catch the random stuff?

[sledgehammer_999]

OK tested it with a "*)" pattern in the case statement. Nothing happened. It was empty.
I would like for a way to parse the "random strings" in any position in the string. Not only at the end.

[littletree76]

A catch-all case within the while loop is used commonly to break out of the loop and no argument will be captured. Thus stream editor (sed) command is required to capture the rest of random stuffs not associated with any valid options defined in the option string "I:D: .....". Using getopts command is a quick/dirty shortcut to parse options/arguments in Unix command line according to Unix POSIX standard, of course it can be implemented in alternative lengthy manner with sed or awk utilities. Unix scripting is rather consistent and flexible.

To have random words to appear anywhere within the whole string, probably I have to implement in different way with string array. I will let you know my finding later on, how much time can you afford to wait ?

[sledgehammer_999]

There isn't a time limit. I was thinking about it the previous days on how to solve the "INCLUDEPATH" problem of the .pro files. Unfortunately pkg-config lumps all cflags together in one variable. So we have to parse it for usage in the INCLUDEPATH, DEFINES and QMAKE_CXXFLAGS.
So this is for v3.2.0 and I am in no hurry to release it. (I still have to do v3.1.10).
Thanks for the cooperation.

[littletree76]

I have revamped the Bash shell script to use just awk command for everything instead of relying on getopts command. Critical section and execution log shown below. The revamped script has been uploaded to my Dropbox site as well as attached to this post.

Wangs-iMac:~$ cd project/qbittorrent
Wangs-iMac:~/project/qbittorrent$ ./parse

CFLAGS = "xxx -I/usr/local/include/libtorrent -DRANDOM_DEF yyy -I/usr/local/include/boost -DRANDOM_DEF2=456 zzz"

VAR1 = "/usr/local/include/libtorrent /usr/local/include/boost"
VAR2 = "RANDOM_DEF RANDOM_DEF2=456"
VAR3 = "xxx yyy zzz"

Wangs-iMac:~/project/qbittorrent$

Code: Select all

awk=/usr/bin/awk
tee=/usr/bin/tee

### Assigment of variable to be processed.

CFLAGS='xxx -I/usr/local/include/libtorrent -DRANDOM_DEF yyy -I/usr/local/include/boost -DRANDOM_DEF2=456 zzz'


### Processing start here.
### Cut and paste between the two dotted comment lines.

### -------------------------------------------------------------------------------------------------------------

VAR1=`echo $CFLAGS | $awk -F" " '{ for ( i=1; i<=NF; i++ ) if ( $i ~ /^-I/ ) printf "%s ", $i }'`
VAR2=`echo $CFLAGS | $awk -F" " '{ for ( i=1; i<=NF; i++ ) if ( $i ~ /^-D/ ) printf "%s ", $i }'`
VAR3=`echo $CFLAGS | $awk -F" " '{ for ( i=1; i<=NF; i++ ) if ( $i !~ /^-I/ && $i !~ /^-D/ ) printf "%s ", $i }'`

### Strip away extra characters.

VAR1=`echo $VAR1 | $awk '{ gsub("-.| $", "", $0); print }'`
VAR2=`echo $VAR2 | $awk '{ gsub("-.| $", "", $0); print }'`
VAR3=`echo $VAR3 | $awk '{ gsub(" $", "", $0); print }'`

### -------------------------------------------------------------------------------------------------------------

[littletree76]

I have come out with final version of the bash script and it is more elegant with function definition though more lengthy. Basically the function extract involves one awk program to process everything and it can be invoked again and again in any script to process any input strings with any option characters. Tricky parts in writing the function include passing variables in/out of awk program within the function. The new parse Bash shell script is attached with this post and uploaded to my Dropbox site.

Wangs-iMac:~$ cd project/qbittorrent
Wangs-iMac:~/project/qbittorrent$ ./parse

CFLAGS = "xxx -I/usr/local/include/libtorrent -DRANDOM_DEF yyy -I/usr/local/include/boost -DRANDOM_DEF2=456 zzz"

VAR1 = "/usr/local/include/libtorrent /usr/local/include/boost"
VAR2 = "RANDOM_DEF RANDOM_DEF2=456"
VAR3 = "xxx yyy zzz"

Wangs-iMac:~/project/qbittorrent$

Code: Select all

### Assigment of variable to be processed.

CFLAGS='xxx -I/usr/local/include/libtorrent -DRANDOM_DEF yyy -I/usr/local/include/boost -DRANDOM_DEF2=456 zzz'

### Function to extract all arguments of the same type given as parameter to the function.
### First parameter is option character and second parameter is the input string.
### Example: extract "-I" "$CFLAGS"
### Function returns resultant string.

extract() {

	echo $2 | awk '

	BEGIN { FS = " " }
		{
			if ( '$1' !~ /^-./ ) {

				for ( i = 1; i <= NF; i++ ) if ( $i !~ /^-./ )
					OUT = OUT " " $i

			} else {

				for ( i = 1; i <= NF; i++ ) if ( $i ~ /^'$1'/ ) {
					sub( "^-.", "", $i)
					OUT = OUT " " $i
				}
			}
		}
	END { sub( "^ | $", "", OUT); print OUT }
	'
}


### Calling the function to process any input strings (must be quoted to avoid interpretation by the shell) with any option characters.
### Note for arbitary arguments without option character, use a big X alphabet as first parameter of the function.

VAR1=`extract "-I" "$CFLAGS"`
VAR2=`extract "-D" "$CFLAGS"`
VAR3=`extract "X" "$CFLAGS"`
[code]


 

[sledgehammer_999]

I wonder if it is possible to auto-create the 3 variables like m4 scripts do in configure.ac.
eg Let's say that you pass only CFLAGS to extract(). extract() parses $1 and creates these 3 vars: $1_INCLUDEPATH, $1_DEFINES and $1_OTHER. So if you pass CFLAGS you get CFLAGS_INCLUDEPATH, CFLAGS_DEFINES and CFLAGS_OTHER.
Is it that possible? I don't if bash has similar feature to the m4 language.

[littletree76]

I don't quite make up what you mean. To generate three variables out of one variable passed to a function is just a matter of packaging in the function, it can be readily implemented by moving the three variables into function definition.

One issue puzzling me is how the function going to know the name beside the value of variable passed to it, so that the three variables generated within the function can have names with the name of passed variable as prefix ? Namely if a variable named CFLAGS is passed to the function, it is straight forward for the function to generated three new variables from value of the CFLAGS variable, but the function has to name the three new variables as : CFLAGS_INCLUDEPATH, CFLAGS_DEFINES and CFLAGS_OTHER within the shell script where it is invoked ?

Unless the function is invoked with three instead of two parameters: extract "-I" "$CFLAGS" "CFLAGS" where $CFLAGS is a variable holding string value and "CFLAGS" is a constant string ?

Take note also most Unix shells (Bash, Korn, Bourne, C) work almost the same way as they have to adhere to POSIX standard, only minor differences with extra features among these shells. C shell claimed to be more user friendly but it is not as consistent as other shells in term of syntax and functionalities of its built-in commands. Thus if you can avoid these extra features (hence minor differences), shell script can be written in rather portable/universal way with no regard to type of shell (lowest denominator is Bourne Shell as it is father of all newer shells).

m4 is a macro processor with parser much more capable than most parsers built into Unix shells as its only purpose is to process input text files to generate output files. Whereas parsers built into Unix shells are just to interpret instructions supplied by shell script file or users through interactive shell.

[sledgehammer_999]

How about this. Let's say that we know all 3 var names before hand. Can we just make the function extract() take only one parameter(CFLAGS) and fill out the 3 in one go? (assuming we hardcode the var names).

[littletree76]

The original function extract has been nested within main function categorise which provide three pre-defined variables after processing: ALL_INCLUDEPATHS, ALL_DEFINES and ALL_OTHERS. This version is more compact than previous version with same functionality. Feel free to add more categories (e.g. ALL_LIBS) and modify default names of pre-defined variables as you see fit. Take note that all functions and main script share one name space thus all pre-defined variables must have unique names.

Wangs-iMac:~/project/qbittorrent$ ./parse

CFLAGS = "xxx -I/usr/local/include/libtorrent -DRANDOM_DEF yyy -I/usr/local/include/boost -DRANDOM_DEF2=456 zzz"

ALL_INCLUDEPATHS="/usr/local/include/libtorrent /usr/local/include/boost"
ALL_DEFINES="RANDOM_DEF RANDOM_DEF2=456"
ALL_OTHERS="xxx yyy zzz"

Code: Select all

CFLAGS='xxx -I/usr/local/include/libtorrent -DRANDOM_DEF yyy -I/usr/local/include/boost -DRANDOM_DEF2=456 zzz'

### Main function returns resultant strings in three pre-defined string variables.
### For nested function, first parameter is option character and second parameter is input string.
### Example: extract "-I" "$CFLAGS"

categorize() {

	extract() {

		echo $2 | awk '

		BEGIN { FS = " "; OUT="" }
			{
				if ( '$1' !~ /^-./ )
					for ( i = 1; i <= NF; i++ ) { if ( $i !~ /^-./ )  OUT = OUT " " $i }
				else
					for ( i = 1; i <= NF; i++ ) { if ( $i ~ /^'$1'/ ) OUT = OUT " " $i }
			}
		END   { gsub( "^ |-.| $", "", OUT); print OUT }
		'
	}
	
	ALL_INCLUDEPATHS=`extract "-I" "$*"`
	ALL_DEFINES=`extract "-D" "$*"`
	ALL_OTHERS=`extract "X" "$*"`
}

### All processing is done with one function call in main script.
### It is not necessary to enclose variable within quotes in command line (refer to "$*" in categorize function).

categorize $CFLAGS

[sledgehammer_999]

Hmm, I was thinking of hardcoding them inside extract() and not wrapping it inside another function(categorize()). I suppose the limitation is that you have to call awk 3 times, one for each category. If that is the case then I'll just use this version: http://qbforums.shiki.hu/index.php/topi ... l#msg12177

[pmzqla]

Maybe this is ugly, but quite compact:

Code: Select all

CFLAGS='-I/usr/local/include/libtorrent -DRANDOM_DEF -I/usr/local/include/boost -DRANDOM_DEF2=456 -Axxx -Byyy -Czzz'
eval $(echo " "$CFLAGS | sed -e 's: -I:\nVAR1=$VAR1\\ :g' -e 's: -D:\nVAR2=$VAR2\\ :g' -e 's: -:\nVAR3=$VAR3\\ -:g')

"sed" simply translates " -I" to "\nVAR1=$VAR1\ ", " -D" to "\nVAR2=$VAR2\ " and " -" to "\nVAR3=$VAR3\ ".

The output of the echo command above, evaluated by eval, is something like the following:

Code: Select all

VAR1=$VAR1\ /usr/local/include/libtorrent 
VAR2=$VAR2\ RANDOM_DEF 
VAR1=$VAR1\ /usr/local/include/boost 
VAR2=$VAR2\ RANDOM_DEF2=456 
VAR3=$VAR3\ -Axxx 
VAR3=$VAR3\ -Byyy 
VAR3=$VAR3\ -Czzz

It is assumed that every flag starts with " -" (note the whitespace. A whitespace is added before the first flag). The whitespace allows to have directories with a dash in their name, but the command might fail in case a directory contains " -D" " -I" or " -" (note the whitespaces) in its name.

EDIT:
Since I don't like eval, here a different solution

Code: Select all

#!/bin/sh

# $1: String to parse
# Set $DEFINES, $INCLUDES, $OTHER
extract() {
  if [ -z "$1" ]; then
    echo "Input string required"
    return 1
  fi

  # Convert " -" to "\n" if not between quotes
  string=$(echo " "$CFLAGS | sed -e 's: -:\n:g' -e 's:"\(.*\)\n\(.*\)":\"\1 -\2":g' -e "s:'\(.*\)\n\(.*\)':\'\1 -\2':g")
  SAVEIFS=$IFS
  IFS=$(printf "\n\b")
  for i in $string; do
    case "$(echo "$i" | cut -c1)" in
      '') ;;
      D) DEFINES="$(echo $i | cut -c2-) $DEFINES";;
      I) INCLUDES="$(echo $i | cut -c2-) $INCLUDES";;
      *) OTHER="-$i $OTHER";;
    esac
  done
  IFS=$SAVEIFS
}